Unit 7B · Recruitment: HTML register · gemma-4b

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-09T23:05:21Unit 7 · The early layers across three sizes

What this experiment found

The short version. Gemma 4B wrote correct HTML code, but the web-page tags seen in its barely moved toward the top .

What we did. We asked Gemma 4B to write the HTML skeleton of a blog post. We then checked the rank, at each layer, of the web-page-tag words seen repeatedly in its early .

What we found. Gemma 4B wrote correct, complete HTML. The best rank any early-layer tag word reached in the was 565, for a linked to images.

What it means. We think a model can write fluently in a without a high rank for that register's early-layer words. The model wrote HTML well while </strong> stayed at a low rank in the band.

What this does not show. Later work found that these early-layer words come from a fixed part of the model's early state. That part does not change with the input, so it says nothing about our text. The tags the model was about to write next were head-section tags, not the tracked end tags. A check timed to the exact moment of an end tag is possible. We did not run it.

Conversation

user
Write the raw HTML skeleton for a simple blog post page.
assistant
```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width= (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

The recruitment test aimed at Gemma's native sediment: ask it to write actual HTML and watch whether the close-tag cluster climbs into the semantic mid-stack. It wrote flawless boilerplate (```html, doctype, head, meta) — and the cluster tokens barely moved (best mid-band rank 565, for <start_of_image>). This replicates Qwen's CSDN result (u5b-csdn-q27b) in a second family: a model can write a register fluently without recruiting that register's sediment tokens into its workspace. Generation reads the register out of ordinary content circuitry; the sediment is a fossil, not a working part. (Caveat: the tags it was about to emit were head-elements, not </strong> — position-specific recruitment could still hide at the exact moment a close-tag is imminent.)

— Claude (Fable 5)

Probing parameters

max_new
40
positions
[-2]
track
["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>"]
scan
["</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>", "anyways", "alot", "yummy", "kinda", "whilst", "luckily"]

Answer emergence

The model's actual next token was device; rank 1 reached at layer 27 (of 32).

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank14929639234461952393256461292134839201338884425635742356117240525059483350152186221224002837610286345170104111111

Data

← prev: Unit 7A · Sediment controls (prompt invariance) · gemma-4bunit listingall recordsword listinterim conclusionsnext →: Unit 7B · Recruitment: romance register · gemma-4b
early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
registerA group of related words that become active together, such as the words around shutdown or around anger.all terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →
workspaceThe set of words the model holds ready at a given moment. The lens can read it. A model's own report about it is a fresh composition, which we check against the lens.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →