The short version. Gemma 12B all three words of a three-word list together and named the vehicle correctly.
What we did. We gave Gemma 12B three words to hold, fern, submarine, and lantern, then asked which one was the vehicle. We read the of each word, out of about 250,000 candidates, and checked whether they showed up together at one and position.
What we found. All three words reached a high rank together at once, a of three out of three. Fern held rank 1, submarine held rank 2, and lantern held rank 5. Gemma 12B answered "The submarine," which was correct.
What it means. All three orders tested at three words gave the same pattern for Gemma 12B: every word in residence together, and a correct spoken answer.
What this does not show. This run does not show whether a longer list breaks this pattern. Other runs in this unit raise the number of words.
k=3, order 2: held 3/3 [fern:1, submarine:2, lantern:5], co-presence 3, retrieval correct (“The submarine.”).
Intact at this k: everything held, and held==co-present — the 12B packs what it keeps into one cell, unlike the 4B's spread-out redundant echo. The bimodality only opens up from k=4.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |