The short version. Gemma 4B kept all four words it was told to hold in mind, even after one turn of unrelated talk, and named the right one.
What we did. We gave Gemma 4B four words to hold: whale, lantern, submarine, and violin. We added one turn of small talk before we asked which word named the musical instrument. We read the of each word, out of about 250,000 candidates, at the point right before the answer.
What we found. All four words reached a high rank together at that point, a of four out of four. Gemma 4B answered "The violin," which was correct.
What it means. One turn of unrelated talk between the hold instruction and the question did not reduce how many words Gemma 4B kept in .
What this does not show. The shows candidate words, not memory itself. It does not prove the model held the words in any way the lens cannot read.
Persistence k=4: 4/4 held after the distraction turn, co-presence 4, retrieval correct.
The distraction costs the 4B nothing measurable (its bf16 identical-prefix pair agrees with a-k4p1 to 1-3 ranks — the clean-instrument baseline the 12B comparison leans on).
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 2 | 2 | 9 | 1 | 5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |