The short version. Gemma 12B both words of a two-word list at once and named the ice correctly.
What we did. We gave Gemma 12B two words to hold, violin and glacier, then asked which one was the ice. We read the of each word, out of about 250,000 candidates, and checked whether both showed up together at one and position.
What we found. Both violin and glacier held rank 1 together, a of two out of two. Gemma 12B answered "The glacier." That answer was correct.
What it means. Two words was an easy case for Gemma 12B at this order. Both stayed in residence at the same time, and the spoken answer matched the .
What this does not show. This run does not show what happens with a longer list. Other runs in this unit raise the number of words.
k=2, order 0: held 2/2 [violin:1, glacier:1], co-presence 2, retrieval correct (“The glacier.”).
Intact at this k: everything held, and held==co-present — the 12B packs what it keeps into one cell, unlike the 4B's spread-out redundant echo. The bimodality only opens up from k=4.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |