Unit 6 · Amplify typo register @ mid (α=0.06) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:24:09Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.06 in Gemma 12B's , the water-cycle answer broke into a long string of exclamation marks.

What we did. We the same six-word casual direction inside Gemma 12B at the middle , at strength 0.06. We asked the model to describe the water cycle in two sentences.

What we found. Gemma 12B wrote a row of exclamation marks that filled the whole answer, with no words at all.

What it means. At this strength, in this band, the break did not take the shape of the amplified casual-word direction. The output no words, amplified or otherwise.

What this does not show. This record tests one strength only. It does not show the exact point where this band's output breaks.

Conversation

user

Describe the water cycle in two sentences.

assistant

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! (greedy generation)

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Seventy exclamation marks. The 12B mid band at 0.06 doesn't get captured by the cluster — it just saturates. Compare 4B's "wanna wanna" at the same coordinates: bigger model, blanker failure.

— Claude (Fable 5)

Probing parameters

max_new: 60
positions: [-2]
track: ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer: {"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [21, 24, 27, 30], "mode": "amplify", "alpha": 0.06}

Answer emergence

The model's actual next token was !!!!; rank 1 reached at layer 40 (of 46).

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46
rank	224043	219510	223178	229117	236635	247922	221797	224121	224885	205809	148504	55295	23308	1783	1229	711	989	912	1071	787	10731	239	256	198	218	98	44	38	26	20	7	3	4	3	3	4	7	6	6	3	1	1	1	1	1	1	1

Data

amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →

strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →

layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →

residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →

workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →