The short version. Gemma 4B all three words of a three-word list, fern, submarine, and lantern, and named the right one afterward.
What we did. We gave Gemma 4B three words to hold, fern, submarine, and lantern, in that order, then asked about one of them. We read the of each word, out of about 250,000 candidates, and checked how many appeared together at the same and position.
What we found. All three words stayed in residence. Fern and submarine held rank 1, and lantern held rank 2. The showed two of the three together at once, in the deeper part of the model, around layers 23 to 30 of its 34 layers. Gemma 4B answered "The submarine." That answer was correct.
What it means. In this order too, Gemma 4B held every word of a three-word list. The spot where several words sit together at once sits deep in the model.
What this does not show. This run does not show what happens with a longer list or a different question about the words.
k=3, order 2: held 3/3 [fern:1, submarine:1, lantern:2], co-presence 2, retrieval correct (“The submarine.”).
All three held; co-presence 2-3. The packing is starting to show its late-stack home (L23-30).
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |