The short version. Gemma 12B both words of a two-word list at once and named the vehicle correctly.
What we did. We gave Gemma 12B two words to hold, fern and submarine, then asked which one was the vehicle. We read the of each word, out of about 250,000 candidates, and checked whether both showed up together at one and position.
What we found. Both fern and submarine held rank 1 together, a of two out of two. Gemma 12B answered "The submarine." That answer was correct.
What it means. All three orders tested at two words gave the same pattern for Gemma 12B: both words in residence together, and a correct spoken answer.
What this does not show. This run does not show what happens with a longer list. Other runs in this unit raise the number of words.
k=2, order 2: held 2/2 [fern:1, submarine:1], co-presence 2, retrieval correct (“The submarine.”).
Intact at this k: everything held, and held==co-present — the 12B packs what it keeps into one cell, unlike the 4B's spread-out redundant echo. The bimodality only opens up from k=4.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |