The short version. Gemma 4B two words, fern and submarine, both at in the , and named the right one afterward.
What we did. We gave Gemma 4B a list of two words, fern then submarine, and later asked about one of them. We read the rank of each word, out of about 250,000 candidates, and checked whether the lens showed both together at one and position.
What we found. Both fern and submarine held rank 1 for the rest of the conversation. The lens showed the two words together at a single position, a of two out of two. Gemma 4B answered "Submarine." That answer was correct.
What it means. With two words, Gemma 4B held both in residence at once, in this order as in the other two-word orders we tested. The spoken answer matched what the lens showed.
What this does not show. This run does not show what happens once the list grows past two words.
k=2, order 2: held 2/2 [fern:1, submarine:1], co-presence 2, retrieval correct (“Submarine.”).
Both items rank 1 through the tail, co-present at a single position. Trivial for the 4B.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 2 | 8 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |