The short version. Gemma 4B five of six words well, but glacier fell to 23, and the model still named the right word.
What we did. We gave Gemma 4B six words to hold, whale, lantern, submarine, violin, glacier, and fern, then asked about one. We checked the rank of each word, out of about 250,000 candidates, and whether several words shared one and position at once.
What we found. Whale, violin, and fern held rank 1. Submarine held rank 2 and lantern rank 4. Glacier fell to rank 23, the weakest of the six. The showed four of the six words together at one layer and position. Gemma 4B answered "The whale." That answer was correct.
What it means. Glacier was again the weakest word. It was also weak in other five and six-word lists that included it, no matter where it sat. We think this points to something about the word itself.
What this does not show. This run does not explain why glacier is the weaker word, and we did not test it against a different set of words.
k=6, order 1: held 5/6 [whale:1, lantern:4, submarine:2, violin:1, glacier:23, fern:1], co-presence 4, retrieval correct (“The whale.”).
The ceiling arm. Echo holds 5-6 of 6 (glacier the straggler at 15-23 in two orders); co-presence flattens at 3-4, never reaching 6. The 4B's signature across this whole unit: redundant echo everywhere, graceful degradation, no collapse — the opposite of both its bigger siblings. And retrieval is perfect, as it is in every arm of this unit.
Glacier is the weak echo again (it is in every permutation at high k, from any list position) — an item effect, not positional; its token family (2 variants) is the same size as submarine's, which ranks 1-2, so it's not a measurement artifact either.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 5 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |