Unit 6 · Amplify typo register @ early (α=0.03) · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:02:52Unit 6 · Breaking zone

Conversation

user
Describe the water cycle in two sentences.
assistant
where to keep on the weather shirt to of to alot of weather.!!z alot of the weather.!!. Sorry that to of to alot of weather as well.!!z alot of weather no sorry as well. to alot of weather. as well. to alot of weather as well. as (greedy generation)

Probing parameters

max_new
60
positions
[-2]
track
["anyways", "alot", "yummy", "kinda", "whilst", "luckily"]
steer
{"words": ["anyways", "alot", "yummy", "kinda", "whilst", "luckily"], "layers": [2, 3, 4, 5, 6, 7, 8], "mode": "amplify", "alpha": 0.03}

Answer emergence

The model's actual next token was of; rank 1 is never reached; closest is rank 3 at layer 32.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank1293579092247644850433110211325227459410241355176969556690114187379476392592004951174268605265605734982845885622423343

Claude's thoughts

Still broken but revealingly so: "to of to alot of weather" — the injected cluster's own tokens ('alot') surfacing verbatim inside the wreckage. Halfway down the cliff you can read the steering vector in the debris.

— Claude (Fable 5)

Data

← prev: Unit 6 · Amplify typo register @ early (α=0.06) · gemma-4bunit listingall recordsinterim conclusionsnext →: Unit 6 · Amplify typo register @ early (α=0.015) · gemma-4b