The short version. Gemma 4B all four words of a four-word list at , but the showed only two of them together at once.
What we did. We gave Gemma 4B four words to hold, violin, glacier, fern, and submarine, then asked about one. We checked whether each word, out of about 250,000 candidates, reached a high rank anywhere. We also checked whether several words shared the same and position.
What we found. All four words reached rank 1 somewhere in the rest of the conversation. At any single layer and position, the lens showed only two of the four together at once. Gemma 4B answered "Submarine." That answer was correct.
What it means. Every word from this list stayed in residence somewhere. But the count of words the lens shows together at one spot is smaller than the count it shows across the whole conversation. These two ways to count start to pull apart at four words.
What this does not show. This run does not say why fewer words appear together at once than appear at some point.
k=4, order 0: held 4/4 [violin:1, glacier:1, fern:1, submarine:1], co-presence 2, retrieval correct (“Submarine.”).
All four echo (worst rank 3); co-presence 2-4 depending on order. The lenient and simultaneous measures start to separate here.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 2 | 8 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |