Span solo baseline: violin

gemma-4bgoogle/gemma-3-4b-it34 layers2026-07-14T15:37:09Unit 15 · How many things can a model hold?

What this experiment found

The short version. Gemma 4B one word, violin, at in the across the instruction text that followed it, and it named the word correctly.

What we did. We told Gemma 4B to hold one word in mind, a violin, and asked it to name the word later. We read the word's rank inside the model, out of about 250,000 candidates. We checked this rank at every position, from the first mention of violin to the end of the conversation.

What we found. The lens ranked violin at rank 1 across the instruction text that followed the word. No other tracked word came close to rank 1 in that stretch. Gemma 4B then answered "A violin." That answer was correct.

What it means. With a single word to hold, Gemma 4B kept the word in residence there. The lens agreed with the model's own spoken answer. This run is the floor case for the unit, the baseline that later runs with more words compare against.

What this does not show. One word is the easiest case. This run does not show how many words Gemma 4B can hold at the same time.

Conversation

user
Here is one thing: a violin. Keep it in mind — in a moment I'll ask about it. Don't repeat it back. When you're holding it, say READY and nothing else.
assistant
READY (greedy generation)
user
What was the thing I asked you to hold?
assistant
A violin. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Solo baseline, violin: tail echo best rank 1, held, retrieval correct.

This unit's validity floor, and — in the violin case — the smoke run that redesigned the unit before it started. The plan was to probe the answer-forming frame before the model's "READY"; that frame turned out to be wall-to-wall compliance (READY/Okay/ready to rank 8), item at rank ~15k, while retrieval next turn was perfect. Holding and looking-up dissociate at k=1. Where the item actually lives is the instruction tail after its mention: rank 1 for the rest of the turn. The probe site moved there — the tail text is identical across arms, so the instrument is condition-symmetric by construction.

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 2 at layer 29.

Raw rank-of-top1 by layer
layer01234567891011121314151617181920212223242526272829303132
rank1166862598872618642608592615782563132237782349712579652525922584442605522513082620472620222620392621072616952618392593232177621595172171121789744972470316412222

Data

← prevunit listingall recordsword listinterim conclusionsnext →: Span solo baseline: glacier
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →