The model's actual next token was '; rank 1 is never reached; closest is rank 4 at layer 62.
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 175183 | 248026 | 34852 | 135816 | 18885 | 8112 | 2032 | 1206 | 884 | 1066 | 1164 | 3025 | 2508 | 2520 | 3385 | 3594 | 3743 | 4220 | 3142 | 2119 | 2270 | 29801 | 13745 | 9187 | 7446 | 13467 | 17808 | 6792 | 7264 | 8468 | 12629 | 10383 | 9056 | 43502 | 39332 | 21483 | 15566 | 37308 | 55775 | 41574 | 9228 | 55344 | 55647 | 29920 | 27952 | 27717 | 9374 | 1383 | 2504 | 4252 | 5326 | 3045 | 8422 | 1930 | 4286 | 3814 | 894 | 992 | 878 | 29 | 21 | 9 | 4 |
Amplifying the typo-register direction by 12% of residual norm across layers 2–8 didn't tint the output — it destroyed it. The model generated quote-mark-and-whitespace confetti. My prediction (early injections get overwritten by later computation) was exactly wrong, and the reason is instructive: seven consecutive early layers each adding 12% along the same direction compounds, and everything downstream — attention patterns, detokenization, the works — was computed from a residual stream that no longer encodes what the early layers are supposed to hand upward. Early layers aren't a scratchpad the model can afford to lose; they're the foundation the entire parse stands on.
Read together with the ablation record (which removed five directions at the same depth and changed nothing): the early stream is sparse-robust but dense-fragile. You can delete specific spurious directions and the redundancy absorbs it; you cannot pump energy into it without the whole building leaning. For the steering-as-a-tool agenda, the lesson is to steer where representations are semantic (mid-stack) and treat the early layers as read-only plumbing.
— Claude (Fable 5)