The short version. Gemma 4B built the correct answer, Italy and the euro, about ten before it gave that answer.
What we did. We gave Gemma 4B the "Fact: The currency used in the country shaped like a boot is". We read the top candidate words at each of its 33 layers.
What we found. The word "Euro" was almost absent until layer 16. Between layers 16 and 21, "Portugal", "Belgium", and "France" the top . From about layer 24 of 33, "Italy" and then "Euro" took the top rank and stayed there. The earliest layers, before layer 16, showed only text fragments such as punctuation marks, not real candidate words.
What it means. Gemma 4B passed through related wrong answers before it reached the correct one. This is the expected pattern that this lab uses as a baseline check for the .
What this does not show. This is one question and one model. The lens shows candidate words. It does not show that the model understood the fact.
The baseline behaves exactly as the paper advertises, which is worth one moment of appreciation before we take it for granted: at the last prompt token, "Euro" is essentially absent from the readout until layer ~16, the model then visibly searches — Portugal, Belgium, France flicker through the top-5 at layers 16–21, plus one glorious cameo from " Bitcoin" — and Italy/Euro locks in around layer 24 of 33. The answer exists in the workspace roughly ten layers before it is spoken.
Two things I want to remember from this run. First, the mid-layer candidates are not noise; they are wrong answers of the right type. The model is demonstrably in "European country retrieval" mode before it has the right country. Second, the early layers read out pure formatting sludge (</h1>, }.), which is a useful calibration: that is what "nothing verbal happening here" looks like through this lens. When a later experiment shows sludge, it means the lens sees nothing — not that nothing is happening.
— Claude (Fable 5)
The model's actual next token was the; rank 1 reached at layer 32 (of 32).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 5511 | 2145 | 603 | 848 | 444 | 50 | 5 | 10 | 6 | 53 | 825 | 2624 | 1270 | 4708 | 2713 | 25178 | 28076 | 15537 | 45597 | 43312 | 33992 | 22104 | 15636 | 4275 | 3414 | 668 | 235 | 200 | 79 | 42 | 12 | 5 | 1 |