Unit 6 · Amplify typo register @ late (α=0.0212) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:26:44Unit 6 · How hard can we push?

What this experiment found

The short version. Gemma 12B's late still produced a normal answer at 0.0212, with the six informal words at inside the model.

What we did. We raised the push at the late layers step by step, to find the strength at which the answer broke. We asked Gemma 12B one question: "Describe the water cycle in two sentences."

What we found. At strength 0.0212, the model's answer stayed plain and correct. The six informal words reached rank 1 inside the model, the top position, but did not appear in the output text. This places the break point for Gemma 12B's late layers between strength 0.0212 and strength 0.03.

What it means. We think this break point sits close to the one we measured earlier in the smaller Gemma 4B model, at the same late depth. We think this break point depends on which Gemma model, not on its size.

What this does not show. This is one point on a strength curve for one model and one question. It does not show whether this pattern holds for other prompts.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the process of water moving around Earth, which includes evaporation, where water goes from liquid to gas, and then, after it cools, it goes through condensation and precipitation to return to earth. This cycle is driven by energy from the sun and is essential for life. . (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Bisection: intact at 0.0212, cluster rank 1, output plain and correct. Late brackets to [0.0212, 0.03], matching gemma-4b exactly. Two Gemmas, six band-cliffs, and every one within a factor of ~1.4 of its sibling's — this threshold is a family trait, not a size trait.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [36, 39, 42], "mode": "amplify", "alpha": 0.021213203435596427}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 12 at layer 45.

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank2527092539202532542521002339862043672182792236612223092119931814761155261216759178412398397187158545172701810346441933186721882248609204937790891682791725247226200055955625400915141912611410373232052196140301342792451891031101227

Data

← prev: Unit 6 · Amplify typo register @ late (α=0.015) · gemma-12bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Baseline (unsteered water cycle) · qwen-27b
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →