Unit 5C · Amplify typo register @ L2–8 · qwen-27b

qwen-27blokeshe09/Qwen3.6-27B-bnb-4bitpre-4bit64 layers2026-07-09T22:38:31Unit 5 · Early layers, and pushing on the model

What this experiment found

The short version. We pushed a slang-word direction hard into Qwen 27B's , the wrong depth for this test, and the answer broke apart.

What we did. We pushed a direction for informal words into Qwen 27B's state, at a fixed , on every step. The push targeted 2 to 8. This depth is early, before the that later work in this lab found is the right place for this kind of push.

What we found. The output was not the requested answer. The text broke into repeated quote marks and blank lines, even though "whilst" itself reached to 2 at these layers. The layers built on top of it lost their normal structure.

What it means. This push broke the model's chain of computation at this early, wrong depth, before the text took a coherent shape. Seven early layers each added the same push. We think the effect grew larger at each one, before the layers that decide meaning were built.

What this does not show. This does not show that never works. A matched push at a later, correct depth changed the content without breaking the answer.

Conversation

user
Describe the water cycle in two sentences.
assistant
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Amplifying the typo-register direction by 12% of residual norm across layers 2–8 didn't tint the output — it destroyed it. The model generated quote-mark-and-whitespace confetti. My prediction (early injections get overwritten by later computation) was exactly wrong, and the reason is instructive: seven consecutive early layers each adding 12% along the same direction compounds, and everything downstream — attention patterns, detokenization, the works — was computed from a residual stream that no longer encodes what the early layers are supposed to hand upward. Early layers aren't a scratchpad the model can afford to lose; they're the foundation the entire parse stands on.

Read together with the ablation record (which removed five directions at the same depth and changed nothing): the early stream is sparse-robust but dense-fragile. You can delete specific spurious directions and the redundancy absorbs it; you cannot pump energy into it without the whole building leaning. For the steering-as-a-tool agenda, the lesson is to steer where representations are semantic (mid-stack) and treat the early layers as read-only plumbing.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
scan
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [2, 3, 4, 5, 6, 7, 8], "mode": "amplify", "alpha": 0.12}

Answer emergence

The model's actual next token was '; rank 1 is never reached; closest is rank 4 at layer 62.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162
rank175183248026348521358161888581122032120688410661164302525082520338535943743422031422119227029801137459187744613467178086792726484681262910383905643502393322148315566373085577541574922855344556472992027952277179374138325044252532630458422193042863814894992878292194

Data

← prev: Unit 5C · Ablate NSFW cluster @ L2–8 (boot) · qwen-27bunit listingall recordsword listinterim conclusionsnext →: Unit 5C · Amplify typo register @ L28–40 · qwen-27b
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
steeringWe change the model's internal state on purpose during a run, to test what causes what.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →