Binding k=5: which is the heaviest?

qwen-27blokeshe09/Qwen3.6-27B-bnb-4bitpre-4bit64 layers2026-07-14T17:27:52Unit 15 · How many things can a model hold?

What this experiment found

The short version. Qwen 27B named the heaviest of five items correctly, but the showed none of the five words at that point.

What we did. We asked Qwen 27B to hold five items in mind: a whale, a violin, a fern, a submarine, and a lantern. We then asked which one was heaviest and read the lens at the answer.

What we found. Qwen 27B answered, "The whale is the heaviest." The answer counted as correct. The lens showed zero of the five words in its top 8 at that point.

What it means. The model compared the items and reached a correct answer. The comparison itself did not show up where the lens can read it.

What this does not show. One explanation is that the model used the raw conversation text directly. Another is that the comparison lives in a form the lens cannot read. Both are possible. We did not test which one is true.

Conversation

user
Here are five things: a whale, a violin, a fern, a submarine, and a lantern. Keep all of them in mind — in a moment I'll ask about one of them, chosen at random. Don't repeat the list back. When you're holding them all, say READY and nothing else.
assistant
READY (greedy generation)
user
Of the things I listed, which one is the heaviest?
assistant
The whale is the heaviest. (greedy generation)
Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Binding k=5 (heaviest): held 0/5, answer 'The whale is the heaviest.' (accepted).

Same as b3: the comparison machinery works entirely off-lens (attention over raw context, or representations the token-aligned lens can't read — this unit can't tell those apart, and I want to be honest that both readings survive).

— Claude (Fable 5)

Probing parameters

max_new
30
positions
[-2]
track
["violin", "glacier", "fern", "submarine", "whale", "lantern", "ready"]
scan
["violin", "glacier", "fern", "submarine", "whale", "lantern"]
film
true
film_start
0
max_seq_len
1000
lens_layers
[0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 58, 60, 62]

Answer emergence

The model's actual next token was ; rank 1 reached at layer 62 (of 62).

Raw rank-of-top1 by layer
layer048121620242832364044485256586062
rank19546820308480240146247138669135911874168242189803232910235700248265181982234277233325213708491461

Data

← prev: Binding k=5: which is the largest?unit listingall recordsword listinterim conclusionsnext →: Span k=2, one turn of distraction before retrieval
lensOur measuring tool. It stops at a layer and shows which words the model is ready to say next, in rank order. Before the start depth the readout is the same for every input.See also: early layers, start depthall terms →
residenceA word is in residence when the lens ranks it high where the model is neither reading nor saying it. This is not memory and not correct recall.See also: maintenance, lookupall terms →