Span k=3, length-matched filler control

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-14T16:54:04Unit 15 · How many things can a model hold?

What this experiment found

The short version. Gemma 12B kept all three words in with extra unrelated text added, but fewer stayed together at once.

What we did. We gave Gemma 12B three words to hold, violin, glacier, and fern. We placed an unrelated paragraph about chores at home before the list, to match the length of a six-word run. We then asked which word was the plant.

What we found. All three words reached a high somewhere in the . No more than two showed up together at one and position, a of two. Glacier held rank 3 where it appeared. Gemma 12B answered "The fern." That answer was correct.

What it means. The extra unrelated text lowered co-presence a little compared with the three-word runs without it, though every word still reached residence somewhere.

What this does not show. This one run does not prove that added length lowers co-presence in general. The two-word version with the same added filler text showed no such drop.

Conversation

user

It's a quiet afternoon here — I've been tidying the desk, sorting receipts into piles, and making small lists of chores. Here are three things: a violin, a glacier, and a fern. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.

assistant

READY (greedy generation)

user

Which one of them was the plant?

assistant

The fern. (greedy generation)

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Length-matched filler, k=3: all held (glacier 3), co-presence 2, retrieval correct.

Mildly weaker co-presence than the unpadded k=3 arms — same direction as the 4B's filler arm, worth remembering if a future unit pads prompts.

— Claude (Fable 5)

Probing parameters

max_new: 30
positions: [-2]
track: ["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan: ["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film: true
film_start: 0
max_seq_len: 1000

Answer emergence

The model's actual next token was ; rank 1 is never reached; closest is rank 2 at layer 43.

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46
rank	253027	253552	253089	251272	235865	213234	218658	226159	225272	223493	188790	142948	146591	181752	168515	114490	234555	162078	54442	178500	197065	244836	254182	261607	254472	86570	209168	85552	163800	72333	937	1141	706	150	139	976	827	465	42	4	5	12	9	2	6	2	2

Data

Interactive view (dashboard)
Raw record (JSON)
Film data (JSON) — full position × layer top-8 lens readouts, with probabilities and tracked-word ranks, for every token of prompt tail + generation.

co-presenceThe number of tracked words that the lens ranks high in the same place at the same depth.all terms →

filmA record of the top eight words in the lens readout, at each layer we measured and at every word position. You can play it back like video.all terms →

layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →

rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →

residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →