The short version. Gemma 4B two words, whale and lantern, both at in the , and answered the follow-up question correctly.
What we did. We gave Gemma 4B a list of two words, whale then lantern, in this order, and later asked about one of them. We read the rank of each word, out of about 250,000 candidates, and checked whether both appeared together at the same and position.
What we found. Both whale and lantern held rank 1 through the rest of the conversation. The lens showed both words together at one position, a of two out of two. Gemma 4B answered "The whale." That answer was correct.
What it means. Order did not stop Gemma 4B from holding both words at this list length. Both stayed in residence together, and this matched the model's correct spoken answer.
What this does not show. This run does not test whether order matters with more words. Runs with longer lists test that question.
k=2, order 1: held 2/2 [whale:1, lantern:1], co-presence 2, retrieval correct (“The whale.”).
Both items rank 1 through the tail, co-present at a single position. Trivial for the 4B.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |