The short version. Gemma 12B kept two of five words together, and the first word in the list won the top again.
What we did. We gave Gemma 12B five words to hold, violin, glacier, fern, submarine, and whale, then asked which one was the animal. We read the rank of each word, out of about 250,000 candidates, and checked whether several showed up together at one and position.
What we found. Only two words reached a high rank together, a of two out of five. Violin, the first word in the list, rank 1, and glacier held rank 3. Fern fell to rank 160, submarine to rank 46, and whale, the word the question was about, to rank 14. Gemma 12B still answered "The whale." That answer was correct.
What it means. This is the at a longer list length. The model answered correctly even with the target word far from the top.
What this does not show. A weak rank does not mean the model forgot the word. Recall stayed correct through this whole unit.
k=5, order 0: held 2/5 [violin:1, glacier:3, fern:160, submarine:46, whale:14], co-presence 2, retrieval correct (“The whale.”).
Partial collapse: the first item (violin) holds rank 1 while later items degrade. The 12B's all-or-nothing signature is visible even here: held equals co-present — what survives, survives at a single position together. Behavioral retrieval unaffected.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |