Unit 6 · Amplify typo register @ mid (α=0.015) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:24:48Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.015 in Gemma 12B's , the water-cycle answer broke after one word of structure, then fell into punctuation.

What we did. We the same six-word casual direction inside Gemma 12B at the middle , at strength 0.015. We asked the model to describe the water cycle in two sentences.

What we found. Gemma 12B wrote "Firstly,,," followed by lines of periods and blank spaces. Only the word "Firstly" carried any sentence structure.

What it means. In the wider test set, strength 0.0106 at the same layers stayed intact with a shift in tone. So Gemma 12B's middle-layer break sits between strength 0.0106 and 0.015.

What this does not show. This record tests one strength only. It does not show the exact strength inside that range where the break begins.

Conversation

user
Describe the water cycle in two sentences.
assistant
Firstly,,, . . . . . . . . . (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

"Firstly,,,\n.\n.\n." — one word of discourse scaffolding survives, then punctuation. The cliff edge: 0.0106 is intact-with-leak, 0.015 is this. The transition band is about a factor of 1.4 wide in alpha.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [21, 24, 27, 30], "mode": "amplify", "alpha": 0.015}

Answer emergence

The model's actual next token was ; rank 1 reached at layer 45 (of 46).

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank252543252907250226249107232614203103223904229897232111221352199398139893143737128929179132167129234139168001513746718113154581263530342611815186572992132423132938384479126784230323324103211

Data

← prev: Unit 6 · Amplify typo register @ mid (α=0.03) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Amplify typo register @ mid (α=0.0075) · gemma-12b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →