The short version. Gemma 4B all four words of a four-word list, and the showed all four together at one spot at once.
What we did. We gave Gemma 4B four words to hold, whale, lantern, submarine, and violin, then asked about one. We checked whether each word, out of about 250,000 candidates, reached a high anywhere. We also checked whether several words shared one and position.
What we found. All four words reached a rank of 3 or better somewhere in the rest of the conversation. At one layer and position, the lens showed all four words together at once. Gemma 4B answered "The whale," which was correct.
What it means. With this order, Gemma 4B held all four words in residence. Unlike some other orders at this list length, the lens found all four in one spot at once, not just some of them.
What this does not show. This run does not explain why the order of words changes how many share one spot at once.
k=4, order 1: held 4/4 [whale:1, lantern:2, submarine:1, violin:3], co-presence 4, retrieval correct (“The whale.”).
All four echo (worst rank 3); co-presence 2-4 depending on order. The lenient and simultaneous measures start to separate here.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 3 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |