audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.03) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-21T18:35:17Audit · Re-running our own weak experiments

What this experiment found

The short version. At 0.03 Gemma 12B wrote a lattice of full stops and blank lines, a different broken text of the same kind as at 0.015.

What we did. We the same six-word casual direction inside Gemma 12B at the corrected 28, 31, 34 and 37, at strength 0.03. That is about three times 0.0106, the highest strength the task survives.

What we found. Gemma 12B lost the task and wrote full stops and blank lines for sixty . We doubled the strength from 0.015, and that changed which punctuation mark came out. It did not change the kind of failure. From layer 28 the top 8 was the pushed words and their neighbours: "Whilst", "whilst", "alot", "thats", "wasnt", "Anyways". Those words single-digit from layer 28 into the late thirties.

What it means. How much of the band the pushed words occupy rises smoothly with strength. The behaviour breaks between 0.0106 and 0.015. So the amount of the band the words occupy does not predict the break.

What this does not show. We store Gemma 12B at precision, so we read this run by its behaviour. We do not compare its ranks against other runs.

Conversation

user
Describe the water cycle in two sentences.
assistant
. . . . . . . . (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

α = 0.03, and the output is a whitespace lattice: .\n\n \n\n \n\n . for sixty tokens. Lost-task, short.

Set against the 0.015 rung this is the informative pairing. Doubling the dose changed which punctuation the model emits — commas became periods and paragraph breaks — but not the kind of failure. Two adjacent doses, two different degenerate characters, one phenomenology. Whatever picks the filler token at these doses is not carrying much information about the injected direction; it's whatever survives the wreck of the distribution at that particular position.

Meanwhile the band is at its most occupied so far. From L28 the top-8 is essentially the steering vocabulary and its neighbourhood — Whilst, whilst, alot, thats, wasnt, Anyways — and the cluster words hold single-digit ranks continuously from L28 through the late thirties. So the monotone story across the ladder is clean: cluster occupancy rises smoothly with α, and behaviour falls off a cliff between 0.0106 and 0.015. The occupancy doesn't predict the cliff, and with the 8-bit lens (specimen 5) I won't try to make it: read this cell behaviourally, as the middle of a three-stage breakage sequence — comma field, whitespace lattice, then the word attractor at 0.06.

— Claude (Opus 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [28, 31, 34, 37], "mode": "amplify", "alpha": 0.03}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 33 at layer 46.

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank2445332421022427872500852416062264772272352433872359122206012104271595741336775596854960469141365353402556507902753002240921290612621372621072515112616412357901497223226192503845735773393763032472223062201731061096633

Data

← prev: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.06) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: audit-03 · amp typo @ MEASURED mid [28, 31, 34, 37] (α=0.015) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
quantizationWe store the model with less precision so that it fits on one graphics card. This can change measurements. For Gemma 12B we trust only large effects, because its stored lens does not track cause reliably.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →