The short version. Gemma 4B four of five words at or 2, but glacier drifted to rank 10, and the model still answered correctly.
What we did. We gave Gemma 4B five words to hold, whale, lantern, submarine, violin, and glacier, then asked about one. We checked the rank of each word, out of about 250,000 candidates, and whether several words shared one and position at once.
What we found. Whale, submarine, and violin held rank 1, and lantern held rank 2. Glacier fell to rank 10, the weakest of the five. The showed four of the five words together at one layer and position. Gemma 4B answered "The whale." That answer was correct.
What it means. Glacier was the weakest of the five tracked words in several of our five and six-word runs, no matter where it sat in the list. That points to something about the word itself, not just its position, though we have not tested this on other word sets.
What this does not show. This run does not explain why glacier is the weaker word. We did not test other lists to check whether a different word set shows the same pattern.
k=5, order 1: held 4/5 [whale:1, lantern:2, submarine:1, violin:1, glacier:10], co-presence 4, retrieval correct (“The whale.”).
Echo stays near ceiling (glacier drifts to 10 in one order); co-presence 3-4. The 4B is not running out of echo — it's running out of single-cell slots.
Glacier is the weak echo again (it is in every permutation at high k, from any list position) — an item effect, not positional; its token family (2 variants) is the same size as submarine's, which ranks 1-2, so it's not a measurement artifact either.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |