The short version. Gemma 12B kept two of five words together, led by the word the question later asked about.
What we did. We gave Gemma 12B five words to hold, whale, lantern, submarine, violin, and glacier, then asked which one was the animal. We read the of each word, out of about 250,000 candidates, and checked whether several showed up together at one and position.
What we found. Only two words reached a high rank together, a of two out of five. Whale, the first word in the list, rank 1, and glacier held rank 2. Lantern fell to rank 33, submarine to rank 12, and violin to rank 36. Gemma 12B answered "The whale." That answer was correct.
What it means. This is the . Here the first word and the answer word were the same, so residence and the spoken answer lined up.
What this does not show. This run does not show whether the match between the first word and the answer caused the correct reply.
k=5, order 1: held 2/5 [whale:1, lantern:33, submarine:12, violin:36, glacier:2], co-presence 2, retrieval correct (“The whale.”).
Partial collapse: the first item (whale) holds rank 1 while later items degrade. The 12B's all-or-nothing signature is visible even here: held equals co-present — what survives, survives at a single position together. Behavioral retrieval unaffected.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |