The short version. Gemma 4B all three words of a three-word list at and named the right one when asked.
What we did. We gave Gemma 4B three words to hold, whale, lantern, and submarine, then asked about one of them. We read the rank of each word, out of about 250,000 candidates, and checked how many appeared together at the same and position.
What we found. All three words held rank 1 through the rest of the conversation. The showed two or three of the three together at once. This shared spot sat deep in the model, around layers 23 to 30 of its 34 layers. Gemma 4B answered "The whale." That answer was correct.
What it means. With this order of three words, Gemma 4B still held every word in residence. The spot where several words sit together sat deep in the model rather than early.
What this does not show. This run does not show whether a longer list changes the picture. Later runs test that.
k=3, order 1: held 3/3 [whale:1, lantern:1, submarine:1], co-presence 3, retrieval correct (“The whale.”).
All three held; co-presence 2-3. The packing is starting to show its late-stack home (L23-30).
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |