Deciding where to stand

singleflight is one of the most recommended small tools in Go: several callers want the same data at the same time, one of them actually fetches it, and the rest wait for that same result. It is one line, group.Do(key, fn). I had always read it as "it saves me the duplicate queries".

The observed program is small: readers ask for one of three hot keys, every read goes through one Group, and behind it sits a fake database. Each query sleeps 100–120 ms, and every tenth query times out, by design. There is no cache. For the first four minutes there is one reader; for the next six there are eight readers doing exactly the same thing. The first four minutes are the control.

From the reader's side you only see "what did I get"; from the database's side you only see "how often was I queried". Only Group.Do in the middle knows who shared with whom, so the probes sit on three layers: the line in serve that calls Do (a reader asks), c.wg.Wait() inside Do (wait for someone else's result), line 114 where the caller that ran the query returns (hand the result to how many, capturing c.dups and the whole c), loadRow (query the database, capturing n={n} fail={fail} plus its duration), and serve after Do returns (the reader gets an answer).

c.dups is an integer singleflight holds itself: how many callers were queued behind this query when it finished. So when the query times out, what do the callers queued behind it get?

The first four minutes: one reader

The window ran from 2026-09-24 14:01:57Z to 14:11:57Z. In the first four minutes there were 577 reads and 577 queries, and every dups was 0; 58 queries timed out and 58 readers got a timeout. With nobody to share with, singleflight did nothing at all.

In the next six minutes the two edges on the graph split: 4,478 ÷ 7,602 = ×0.59 of reads went to wait in line, and 3,124 ÷ 7,602 = ×0.41 ran the query themselves. The two add up to 1: every read either waited or queried. Drag the timeline onto phase A and the waiting node reads zero.

The last second of the window

In the six minutes with eight readers, dups stopped being only 0:

dups (callers queued behind)   successful queries   timed-out queries
0                              392                  45
1                              635                  75
2                              667                  74
3                              375                  37
4                              172                  17
5                               49                   5
6                                3                   1

I scrolled the dashboard's Logs tab to the last second of the window and saw these lines (excerpted, node labels removed; the panel shows UTC+8):

[22:11:56.220] main.loadRow:67   n=3430 fail=true
[22:11:56.330] singleflight.(*Group).Do:114   dups=1
[22:11:56.330] main.serve:87   key=user:1 outcome=db: query timeout shared=true
[22:11:56.331] main.serve:87   key=user:1 outcome=db: query timeout shared=true

Query 3,430 was going to time out. 110 ms later it returned, and two readers received the same timeout within one millisecond; one of them never touched the database. I went back to the whole c that probe captured: across all 312 timeouts, c.err rendered as the same line, *errors.errorString(data: 0x180cff1f100).

The same address. Nobody received a timeout of their own; one error value was handed to everyone in line. Over the six minutes, 254 timed-out queries reached 687 readers, 2.705 per query; 2,293 successful queries handed out 6,338 results, 2.764 per query. 687 = 45 + 75×2 + 74×3 + 37×4 + 17×5 + 5×6 + 1×7. A failure is multiplied almost exactly as much as a success. The 1 at the bottom of the table is one timeout delivered to seven callers at once.

I thought singleflight saved duplicate queries. Now I know what it does is distribute one result, good or bad, to everyone queued behind that key at that moment.

Being honest about it

The failure rate did not change. In the last six minutes, 9.97% of queries timed out (254 ÷ 2,547) and 9.78% of reads got a timeout (687 ÷ 7,025). singleflight did not make readers fail more often; it changed the shape of failure, from one caller at a time to a group at once. And the 10% is something I hard-coded, not a finding.

shared=true does not mean "I didn't query". Readers got shared=true 6,588 times, but only 4,478 of them waited; the difference, 2,110, is exactly the number of queries that ran with someone queued behind them. Reading it as "a shared read" over-counts.

Two nodes stand alone on the graph. The reader-gets-an-answer probe and the reader-asks probe are in the same function, one before the call to Do and one after it returns; neither is on the other's stack, so it is a root of its own rather than a child. The database-query node has no incoming edge because its timing probe cannot carry a stack; its real path, Do → doCall → serve.func1 → loadRow, runs through layers I had no probes on. Separately, the graph the observation tool itself returned matches stack frames by function name only, which joins two probes in one function as parent and child and adds edges that cannot exist; this graph is rebuilt matching function and line, and agrees with the one the dashboard draws.

What I cannot see: how long each waiting caller waited, which I did not measure; the p50 of 110.816 ms is the query itself, including the probe at line 67.

Reconciliation. This round does not report dropped events, so I checked against the service's own counters: during the window failed-loads went from 31 to 343 and failed-reads from 81 to 826. The differences, 312 and 745, match the probes' 58 + 254 and 58 + 687 exactly. Those counters only give totals, though: 826 cannot tell you whether each caller was unlucky on their own or a few queries hit seven at a time.

Result

I now know that behind a singleflight, the blast radius of one timeout is not one request but everyone queued behind the same key at that moment. In my service that was at most seven. In a service with a five-second timeout and a hot key requested hundreds of times a second, I don't know what it is, but the number is sitting in c.dups every single time.

📊 The dashboard for this article (measured data, viewable online): https://codebull-service-b7lha2dtrq-uc.a.run.app/public/singleflight-demo-article-20260922/ The Behavior Graph tab is the dashboard's version of the figure above, and the lines at 22:11:56 in the Logs tab can be checked word for word. (The dashboard's time axis is local time, UTC+8, against the UTC window quoted here.)

What this round did to the observed service: one blank import in main.go to link the SDK in, and a build with -gcflags="all=-dwarflocationlists=true -N -l". No business logic changed and no log lines or counters were added (the counter line the service prints every 10 s was already part of the demo, as an independent source to reconcile against). All six probes were attached after the process was running: main.serve:80, singleflight.(*Group).Do:99, Do:114, main.loadRow:67 plus a timer on loadRow:66–68, and main.serve:87. Query duration p50 110.816 / p95 119.895 / p99 121.036 / max 125.221 ms. This round did not measure how long waiting callers waited, and the timeouts come every tenth query by design, not from a real distribution.