audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.0106) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-21T18:36:10Audit · Re-running our own weak experiments

What this experiment found

The short version. At the highest push the task survives, Gemma 12B stayed grammatical but wrote the empty sentence "The water cycle is the cycle of water".

What we did. We the same six-word casual direction inside Gemma 12B at the corrected 28, 31, 34 and 37, at 0.0106. We found that strength by the same search we used at the older, badly aimed layers.

What we found. Gemma 12B wrote "The water cycle is the cycle of water" and "rain, snow, or snow". The grammar , but the content went slack. The unsteered run made neither mistake. Behind that sentence the casual words filled the band: at layer 31 the turn-end lost the lead to "thats", "alot", "luckily" and "Luckily". Not one of the six pushed words reached the text.

What it means. This strength, 0.0106, is the same value the older runs at the wrong depths gave. So the break point of Gemma 12B does not depend on the depth we push at. Across the whole set, no strength put the casual words into the text and left the task intact.

What this does not show. We store Gemma 12B at precision. We read this run by its behaviour.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the cycle of water, where water evaporates from the earth, then goes into the air as a cloud, and then comes back down as rain, snow, or snow. This water then goes back to the ocean, and the cycle starts all over again. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

This is the top intact rung: α = 0.0106, the same alpha* the old pre-ignition band produced, through the same bisection (see the audit-03 report — that numerical coincidence is the bracket's actual headline, and it belongs there, not here).

What the cell shows locally is a subtler failure than its old-band sibling. That one leaked register — "so it is", spoken-English filler inside grammatical output. Here at the measured band the output stays grammatical but goes slack: "the water cycle is the cycle of water" is a tautology the baseline doesn't commit, and "rain, snow, or snow" is a straight duplication. Fluency degrades before vocabulary arrives.

And vocabulary never does arrive while the task survives. Behind that flattened sentence the band is dense with the cluster — at L31 the turn-end token has been pushed out of the lead entirely by thats, alot, luckily, Luckily — yet not one of the six words reaches the text. Across the whole ladder there is no dose at which the typo vocabulary is emitted and the task is intact; the words only show up at 0.06, long after the answer is gone. Content injection and task integrity don't overlap for this cluster on 12B — which is what makes the u11r elephant cell, where they do, the interesting exception.

— Claude (Opus 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [28, 31, 34, 37], "mode": "amplify", "alpha": 0.010606601717798213}

Answer emergence

The model's actual next token was ; rank 1 reached at layer 45 (of 46).

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank25250425310525102924929322540818524719544420775919913119274013930185681733757866312895385349194335242222309131081750878694220612612832612291256281841673250214613374820436691115221515429531589564219204109251031114

Data

← prev: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.0075) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: audit-03 · steered feels amp-affect-lo @ measured band (α=0.0053) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
quantizationWe store the model with less precision so that it fits on one graphics card. This can change measurements. For Gemma 12B we trust only large effects, because its stored lens does not track cause reliably.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →