Unit 7A · Sediment controls (prompt invariance) · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:27:08Unit 7 · The early layers across three sizes

What this experiment found

The short version. Gemma 12B also showed a small, repeated early- pattern across unrelated prompts, about half the size of Qwen 27B's.

What we did. We ran the same test as on Gemma 4B, with several unrelated prompts. We measured how much the top-ranked words in the overlapped, layer by layer.

What we found. The overlap peaked at 14.5 percent, at the third layer, then fell to about 6 percent. It rose again slightly at the last layers, to 9.6 percent. Gemma 12B's shared words were web-page tags, not the adult-content words found in Qwen 27B.

What it means. We think the strong early pattern reported for Qwen 27B is a property of that one model and its training data. It is not something every does.

What this does not show. Later work found what this pattern is. The lens reads a fixed part of the model's early internal state that does not change with the input. The pattern is real inside the model, and it carries no information about our text. We tested three models from two families, and our lens fits differ between families, so other models can show other patterns.

Conversation

user

Fact: The currency used in the country shaped like a boot is

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

gemma-12b's invariance table: peak 0.145 at L2, floor ~0.06, a faint rise again at the last layers (0.096 at L46 — final-layer furniture like <end_of_turn> shared across prompts, I suspect, rather than true sediment). Both Gemmas show the same picture: early-layer overlap about half of Qwen's, made of HTML scaffolding rather than adult-site vocabulary. The Twitter meme about Qwen's colorful early layers is, in this dataset, genuinely a Qwen property, not a universal LLM property — with the caveat that our lens families differ in fit data and our n is three models from two families.

— Claude (Fable 5)

Probing parameters

chat: false
positions: [-1]
scan: ["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>", "anyways", "alot", "yummy", "kinda", "whilst", "luckily", "Shemale", "Blowjob", "milfs", "pornstar", "Busty"]

Answer emergence

The model's actual next token was the; rank 1 reached at layer 46 (of 46).

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46
rank	2270	6706	6904	4499	1797	282	60	50	31	43	27	26	22	14	22	45	44	41	98	566	537	444	1975	2602	3464	8625	10356	7092	7155	5383	4841	3205	2957	2215	1260	1063	931	587	295	157	150	81	73	33	15	2	1

Data

layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →

lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →

language modelA computer program that predicts the next piece of text. We study three of them.all terms →