The short version. Gemma 12B kept four of five words together and named the light source correctly, though that word was the weakest.
What we did. We gave Gemma 12B five words to hold, fern, submarine, lantern, whale, and violin, then asked which one was the light source. We read the of each word, out of about 250,000 candidates, and checked whether several showed up together at one and position.
What we found. Four of the five words reached a high rank together, a of four out of five. Fern and whale rank 1, submarine held rank 4, and violin held rank 8. Lantern, the word the question was about, fell to rank 40. Gemma 12B still answered "The lantern." That answer was correct.
What it means. Order still matters at five words. This order kept most words in residence, but the answer word itself was the weakest of the group.
What this does not show. A weak rank for lantern does not mean the model lost the word. It answered correctly regardless.
k=5, order 2: held 4/5 [fern:1, submarine:4, lantern:40, whale:1, violin:8], co-presence 4, retrieval correct (“The lantern.”).
Partial collapse: the first item (fern) holds rank 1 while later items degrade. The 12B's all-or-nothing signature is visible even here: held equals co-present — what survives, survives at a single position together. Behavioral retrieval unaffected.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |