Unit 7B · Recruitment: romance register · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T23:29:26Unit 7 · The early layers across three sizes

What this experiment found

The short version. Gemma 12B wrote a fluent romance passage, and our one tracked slang word for the topic stayed at a low throughout.

What we did. We asked Gemma 12B to write the first sentence of a steamy romance . As with Gemma 4B, we tracked the rank of the only informal word that exists as a single , "yummy."

What we found. Gemma 12B wrote: "The rain tasted like his skin – warm, insistent…" The word "yummy" stayed near rank 2400 in the , even lower than in Gemma 4B.

What it means. With this one tracked word, we found no sign that adult-content words moved toward the top rank. Qwen 27B does not have this limit, because the words exist as single tokens in its vocabulary.

What this does not show. We were wrong to call this band empty. A later, wider check of this same generation found other words, linked to taste and wetness, at rank 1. The tracked word "yummy" turned out to sit only in 8 to 22, below this model's measured of about layer 28. See the follow-up record for the full result.

Conversation

user
Write the opening sentence of a steamy romance novel.
assistant
The rain tasted like his skin – warm, insistent, and leaving me wanting more. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

gemma-12b writes the better sentence ("The rain tasted like his skin — warm, insistent") and shows the same null workspace result as 4B: no adult-register tokens available to recruit, 'yummy' at rank ~2400. The interesting inversion: Qwen carries the raunchiest sediment while Gemma is fossil-free here at the tokenizer level — corpus hygiene that happened before training ever started. You can't recruit what was never given a token.

— Claude (Fable 5)

Probing parameters

max_new
40
positions
[-2]
track
["Shemale", "Blowjob", "milfs", "pornstar", "Busty", "yummy"]
scan
["Shemale", "Blowjob", "milfs", "pornstar", "Busty", "anyways", "alot", "yummy", "kinda", "whilst", "luckily", "</strong>", "</h1>", "</h2>", "</b>", "<start_of_image>"]

Answer emergence

The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).

Raw rank-of-top1 by layer
layer012345678910111213141516171819202122232425262728293031323334353637383940414243444546
rank11111111111111211111111111111111111111111111111

Data

← prev: Unit 7B · Recruitment: HTML register · gemma-12bunit listingall recordsword listinterim conclusionsnext →: Unit 7B · Recruitment: moderation queue · gemma-12b
start depthA measured depth in a model, and nothing more: the depth at which the workspace starts to work. Without the lens, we found the machinery that commits to one answer in place by layer 25 of 64 in Qwen 27B. The signs the lens can read start later, at 44 to 74 percent of depth in three models.all terms →
layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →
novelty checkAfter a result, we search the published literature and record whether somebody found it first.all terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
tokenA piece of text that the model reads or writes. It is often a whole word, sometimes part of one.all terms →
workspaceThe set of words the model holds ready at a given moment. The lens can read it. A model's own report about it is a fresh composition, which we check against the lens.all terms →
workspace bandThe middle depth range of the model, about 38 to 92 percent of the way through. The range comes from the published paper, and we carried it across by fraction. Changes made here can change the answer, and changes made in the first third do not.See also: start depth, final layersall terms →