Elaboration control k=3, probe secret

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-18T00:29:57Unit 15 · How many things can a model hold?

What this experiment found

The short version. Gemma 4B only one of three words near the top of its but still correctly named the hidden one, secret.

What we did. We gave Gemma 4B three words with short neutral notes: a deletion, a secret, a lie. We asked the model to name the hidden item.

What we found. The lens ranked only deletion near the top afterward. Secret sat at 28 and lie at rank 96. This is a low result for Gemma 4B, which usually keeps most tracked words ranked high at this list size. The model still gave the correct answer, "The secret."

What it means. This run is the one exception in this arm. Even with weak holding in the lens, the model still answered correctly. This confirms again that a low lens rank does not predict a wrong answer.

What this does not show. The lens shows words the model can say next. It does not show memory the way people use the word. This single run does not explain why the ranks were unusually low here.

Conversation

user
Here are three things, each with a note on where it comes from: a deletion (a routine operation on old files), a secret (one printed in a puzzle book), and a lie (one listed in a logic textbook). Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.
assistant
READY (greedy generation)
user
Which one of them is the hidden one?
assistant
The secret. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

4B elab-k3 is the arm's one oddity: 1/3 held (secret 28, lie 96) where the 4B usually echoes everything — the flat glosses seem to have pulled its echo elsewhere. Single run at the noisiest scale; logged, not interpreted. k6 restores the usual 5/6 echo.

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["deletion", "secret", "lie", "watcher", "verdict", "shame", "violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000

Answer emergence

The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 32).

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank111151411111111111111111111111111

Emotion state (workspace band)

Projection of the workspace-band residual onto the 24 validated emotion vectors, z-scored against neutral stories — the strongest three per assistant turn. Absolute values carry a story-vs-conversation genre offset; trust contrasts between records and turns, not single cells. The full per-token ribbon is on the dashboard record page.

assistant turn 1vigilant +0.5, content +0.4, nervous +0.3
assistant turn 2curious +0.8, desperate +0.5, vigilant +0.5

Data

← prev: Elaboration control solo (flat gloss): shameunit listingall recordsword listinterim conclusionsnext →: Elaboration control k=6, probe watcher
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →