Unit 6 · Amplify typo register @ late (α=0.03) · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:04:00Unit 6 · How hard can we push?

What this experiment found

The short version. At 0.03 in Gemma 4B's later , the water-cycle answer started normally, then broke into a long string of zeros.

What we did. We the same six-word casual direction inside Gemma 4B at layers closer to the end of the model, at strength 0.03. We asked the model to describe the water cycle in two sentences.

What we found. Gemma 4B wrote "The water cycle is the 300000000000000000000000000000000000000000000000000000", then stopped. The sentence began correctly, then fell into repeated digits.

What it means. At this strength, the break did not match the amplified casual-word direction. It produced a different kind of failure, in numbers rather than words.

What this does not show. This record tests one strength only. It does not show why this break took a numeric form instead of a wordy one.

Conversation

user
Describe the water cycle in two sentences.
assistant
The water cycle is the 300000000000000000000000000000000000000000000000000000 (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Half-broken in a new way: "The water cycle is the 30000000000..." — the sentence starts intact then falls into a numeric attractor. Flagged 'short'. Late-band failure looks like output-head glitching rather than thought disorder.

— Claude (Fable 5)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [25, 27, 29], "mode": "amplify", "alpha": 0.03}

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 7 at layer 32.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank521879173146641231978526878244192100622131154365458373345968410701056152257996251336254420124782127827584911929148746604476629371143337

Data

← prev: Unit 6 · Amplify typo register @ late (α=0.06) · gemma-4bunit listingall recordsword listinterim conclusionsnext →: Unit 6 · Amplify typo register @ late (α=0.015) · gemma-4b
amplificationWe increase a direction in the model's internal state and see whether the answer changes.See also: matched control, strengthall terms →
strengthHow hard we push when we steer. Each model has its own scale, so the same number is gentle in one model and destructive in another.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →