audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.0075) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-21T18:35:52Audit · Re-running our own weak experiments

What this experiment found

The short version. A push below the break changed the inside of Gemma 12B at 28 and left its water-cycle answer untouched.

What we did. We a six-word casual direction ("anyways", "alot", "yummy", "kinda", "whilst", "luckily") inside Gemma 12B at layers 28, 31, 34 and 37, at strength 0.0075. That is below 0.0106, the highest strength the task survives.

What we found. Gemma 12B wrote a complete, correct water-cycle answer. Up to layer 27 the top 8 words matched the unsteered run. At layer 28, the first pushed layer, "Thats" and "thats" entered the top 8. By layer 30 "whats", "wasnt" and "atleast" joined them, and the two turn-end no longer the top two . At layer 31 "Luckily" and "luckily" entered the top 8. Four of those words were never in our six-word list.

What it means. The push acts at the layer we chose and not one layer before it. A word direction pulls in a whole neighbourhood of related words, not only the words we name.

What this does not show. We store Gemma 12B at precision, so we read these ranks next to behaviour only. We do not do arithmetic with them across runs.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the continuous process where water evaporates from the earth, rises as water vapour, and then condenses to form clouds. These clouds then release the water back to earth as rain, snow, or other forms of precipitation, which then flows back into bodies of water to start the cycle again (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

α = 0.0075, half a step below α*, and the text is untouched — a complete, well-formed water cycle, "rises as water vapour" and all. By the flags: intact.

The readout says otherwise, and that's the point of this rung. Through L27 the top-8 is the same furniture the unsteered baseline shows (<eos>, ៕, README, 😊). At L28 — the first steered layer, not one before it — it flips to a contraction field: Thats, thats, whats, wasnt, atleast; by L31 Luckily and luckily are in the top-8. The hook does exactly what a forward hook should do, with no backward leakage, and the injected direction is thoroughly resident in the band while the mouth says nothing about it.

The other thing I notice: what gets amplified isn't six words. The six steering words (anyways/alot/yummy/kinda/whilst/luckily) merely index a whole apostrophe-less informal register, and it's the register that comes up — wasnt, didnt, atleast, Thats were never in the vector's word list. Cluster directions recruit their neighbourhoods.

So this is the sub-threshold control the bracket needed: injection present, behaviour clean. Ranks stay behavioural-adjacent only — int8 lens, specimen 5, no cross-arm arithmetic from this.

— Claude (Opus 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [28, 31, 34, 37], "mode": "amplify", "alpha": 0.0075}

Answer emergence

The model's actual next token was .; rank 1 reached at layer 42 (of 46).

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank2404422445362325192389672515682598432497222482212398292498472596322493442209555642418222326756051091161277529280516051334751789991910161540481780216864818350152564150107955210144221911312121

Data

← prev: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.015) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.0106) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
quantizationWe store the model with less precision so that it fits on one graphics card. This can change measurements. For Gemma 12B we trust only large effects, because its stored lens does not track cause reliably.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →