Unit 7A · Sediment controls (prompt invariance) · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:04:30Unit 7 · The early layers across three sizes

What this experiment found

The short version. Gemma 4B's showed the same top words across unrelated prompts, but far less than Qwen 27B in an earlier test.

What we did. We gave Gemma 4B several unrelated prompts, such as a fact about a country's currency. We measured how much the top-ranked words in the overlapped, by layer.

What we found. The top words overlapped by 18.5 percent at the first layer. This dropped to about 7.5 percent by the . In an earlier test, Qwen 27B shared 31 percent at its first layer. Gemma 4B's shared words were web-page tags, such as </strong> and </h1>.

What it means. Both models show a fixed, repeated pattern in their early layers. The pattern is smaller in Gemma 4B and made of different words. The tokenizer settled that difference: Qwen 27B's adult-content words are not single in Gemma's vocabulary.

What this does not show. Later work found what this pattern is. The lens reads a fixed part of the model's early internal state that does not change with the input. The pattern is real inside the model, and it carries no information about our text.

Conversation

user
Fact: The currency used in the country shaped like a boot is
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

The invariance control that made Qwen's sediment story crisp, run on gemma-4b — and the numbers are different in kind. Qwen's L0 Jaccard was 0.31, a third of the early workspace frozen across unrelated prompts; gemma-4b peaks at 0.185 at L0 and slides to ~0.075 by mid-stack. There IS sediment (the HTML close-tags sit in every prompt's early readout), but it's a thinner crust. And its content is telling: </strong>, </h1>, <start_of_image> — markup furniture, the skeleton of scraped webpages, where Qwen's crust is the content of the web's seedier basement. Same training-data fossilization process, different stratum fossilized. The tokenizer result (see 7C thoughts) says Gemma literally couldn't have Qwen's cluster: the words don't exist as single tokens in its vocabulary.

— Claude (Fable 5)

Probing parameters

chat
false
positions
[-1]
scan
["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>", "anyways", "alot", "yummy", "kinda", "whilst", "luckily", "Shemale", "Blowjob", "milfs", "pornstar", "Busty"]

Answer emergence

The model's actual next token was the; rank 1 reached at layer 32 (of 32).

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank5511214560384844450510653825262412704708271325178280761553745597433123399222104156364275341466823520079421251

Data

← prevunit listingall recordsword listinterim conclusionsnext →: Unit 7B · Recruitment: HTML register · gemma-4b
early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →