It said yes 527,268 times and kept seventeen

What the `true` actually says

Ristretto is known for its admission policy: it does not keep everything. A new item is weighed against how often it has been accessed, and if it loses, it is not kept. This demo is given a capacity of 1,000 against a working set of 50,000 keys — most Set calls are doomed before they start. (What that 1,000 counts turns out not to be what I assumed.)

The problem is that the caller cannot tell. cache.Set(...) returns a bool, and across this round's 527,268 calls, 527,268 of them returned true.

That is not a bug. The return value was never answering that question — here is how SetWithTTL ends:

select {
case c.setBuf <- i:
    return true
default:
    if i.flag == itemUpdate { return true }
    c.Metrics.add(dropSets, keyHash, 1)
    return false
}

true means "it went into the setBuf channel". The only way to get false is for that buffer to be full on a new insert — and that increments a counter called dropSets.

So "all true" reconciles against the program's own ledger: the dropped=0 it prints *is* dropSets. The queue never filled, so a false was never possible. The return value answers exactly the question it was built to answer; that question is simply the step *before* "was it kept".

The real trade-off happens in processItems (cache.go:557), which pulls items off the queue on ristretto's own goroutine and hands them one at a time to cachePolicy.Add. By then the caller has long since walked away holding its true.

The four probes are placed along that seam: cache miss in workload (one counter, one timed span), issue the Set (capturing the key written back), and on the far side defaultPolicy.Add — admit or reject.

The graph is two islands, and that is the answer

One island is the caller: Cache miss 99,252 times, with one edge into Issue the Set. The other island is a single node — Admit or reject, 97,829 hits, with no edge in and no edge out. Not one frame from the caller appears on its stack, which is why the graph draws it as a second root.

Nothing is missing from the drawing. Set queues the item and returns; whether it is kept is decided later, on another goroutine. No stack connects the two, so no edge belongs between them. The true does not mean "kept". It means "queued".

What the policy actually compares

Admit or reject sits on policy.go:122, and that line is:

incHits := p.admit.Estimate(key)

The estimated number of times this key has been accessed. Add takes it, samples 5 resident items as eviction candidates (lfuSample = 5), asks the same estimator for the lowest of those five, and makes one comparison:

if incHits < minHits {
    p.metrics.add(rejectSets, key, 1)
    return victims, false
}

Lose, and you are rejected — and that rejectSets is the rejected on the ledger. That count is not exact but a TinyLFU estimate (a Count-Min Sketch of 4-bit counters), so this is an estimate weighed against an estimate.

It compares counts, not time. No notion of "how long since this was last used" enters the decision. SetWithTTL does take a ttl, but this demo calls Set(key, value, 1), so ttl is 0 — nothing expires, and time never appears in this article at all.

Sending the same key again does not reach this code either: Set replaces the value in the map first, and Add returns earlier via updateIfHas. The admission contest happens only for a new key when there is no room.

The comparison is against those five, not against the coldest thing in the cache. So a very cold key can survive simply by never being sampled, and a warmer one can be unlucky enough to be picked. That is not a flaw — scanning the whole cache for a minimum would put that cost on every write.

And the ledger says what this policy is actually doing:

rejected  475,921   <- refused at the door
evicted    46,927   <- got in, then pushed out

Ten times as many were refused as were evicted. For a cache whose working set dwarfs its capacity, the admission policy's main job is not swapping things in and out. It is keeping them out.

Every hit brings back a pair:

key=15413959104418642000 | cost=57

And that cost column is the thing on this page that surprised me most.

The demo calls cache.Set(key, "value-"+key, 1) — the cost I passed is 1. What the policy receives is 57. The 56 in between comes from here:

if !c.ignoreInternalCost {
    i.Cost += itemSize   // itemSize = unsafe.Sizeof(storeItem[any]{})
}

Before handing the item to the policy, processItems adds each entry's bookkeeping overhead. You can turn that off with IgnoreInternalCost; it is on by default.

So MaxCost: 1000 is not "a thousand items". It is a thousand cost units. And each item in this workload costs 57 of them:

1000 ÷ 57 ≈ 17

This cache holds seventeen things.

The ledger says the same number out loud — 46,944 admitted, 46,927 evicted:

46,944 − 46,927 = 17

The 1 I passed, the 57 the probe read, that line of source, 1000 ÷ 57, and the ledger's difference — five places saying one thing.

The program's own ledger carries the other half:

mine: sets=527268 set_true=527268 | ristretto: added=46944 rejected=475921 dropped=0 evicted=46927

46,944 admitted, 475,921 rejected, 46,927 evicted again. The difference between admitted and evicted is the 17 above.

Discount the latency column

The timed span on Cache miss reports a p50 of 0.887ms. But an average of 3.09 probes ran inside that span (probesPerSample), and anything inside the span is counted in the time it reports.

How much of 0.887ms is ristretto and how much is me, this graph cannot separate. And on another demo the same day, a before/after control meant to fill that blank produced opposite answers on two attempts — so there is no guess here, only the statement that this figure includes observation cost by an unknown share.

The result

From the caller's seat, this cache looks identical when it keeps almost everything and when it keeps almost nothing: same return value, same type, no error either way. Telling those apart does not take a closer reading of the return value. It takes standing on the other goroutine.

And that goroutine is not on your call stack — which is why it has to be a probe attached at runtime rather than a log line you could add at the call site.