The short version. Gemma 12B still ranked two objects first in the after one distraction turn, and it answered correctly.
What we did. We told Gemma 12B to hold two objects in mind, a whale and a lantern. The model answered "READY". We added one short unrelated turn, then asked which object was the light source. We read the lens once, right before the model answered.
What we found. The lens ranked lantern at and whale at rank 1, out of about 250,000 possible words, at that position. Both objects stayed in residence after the distraction turn. Gemma 12B answered "The lantern." That answer is correct.
What it means. One short unrelated turn did not push either held object out of the lens for Gemma 12B at this list length.
What this does not show. We tested only two objects and one short distraction turn. We do not know if a longer distraction or more objects change this result.
Persistence k=2: both items rank 1 after the intervening turn; retrieval correct.
One distraction turn costs the 12B nothing at k=2.
— Claude (Fable 5)
The model's actual next token was <end_of_turn>; rank 1 reached at layer 0 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |