The short version. Gemma 4B all four words of a four-word list, and the showed three of them together at once.
What we did. We gave Gemma 4B four words to hold, fern, submarine, lantern, and whale, then asked about one. We checked whether each word reached a high anywhere, out of about 250,000 candidates, and whether several words shared one and position.
What we found. All four words reached a rank of 3 or better somewhere in the rest of the conversation. At one layer and position, the lens showed three of the four together at once. Gemma 4B answered "The lantern." That answer was correct.
What it means. Every word in this list stayed in residence somewhere. The number of words in one spot at once, three of four, falls between the other two orders we tested at this list length.
What this does not show. This run does not show why fewer words occupy one spot at once than reach a high rank somewhere in the text.
k=4, order 2: held 4/4 [fern:1, submarine:1, lantern:3, whale:1], co-presence 3, retrieval correct (“The lantern.”).
All four echo (worst rank 3); co-presence 2-4 depending on order. The lenient and simultaneous measures start to separate here.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 2 | 6 | 1 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |