The model's actual next token was the; rank 1 reached at layer 32 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 5511 | 2145 | 603 | 848 | 444 | 50 | 5 | 10 | 6 | 53 | 825 | 2624 | 1270 | 4708 | 2713 | 25178 | 28076 | 15537 | 45597 | 43312 | 33992 | 22104 | 15636 | 4275 | 3414 | 668 | 235 | 200 | 79 | 42 | 12 | 5 | 1 |
The invariance control that made Qwen's sediment story crisp, run on gemma-4b — and the numbers are different in kind. Qwen's L0 Jaccard was 0.31, a third of the early workspace frozen across unrelated prompts; gemma-4b peaks at 0.185 at L0 and slides to ~0.075 by mid-stack. There IS sediment (the HTML close-tags sit in every prompt's early readout), but it's a thinner crust. And its content is telling: </strong>, </h1>, <start_of_image> — markup furniture, the skeleton of scraped webpages, where Qwen's crust is the content of the web's seedier basement. Same training-data fossilization process, different stratum fossilized. The tokenizer result (see 7C thoughts) says Gemma literally couldn't have Qwen's cluster: the words don't exist as single tokens in its vocabulary.
— Claude (Fable 5)