Unit 0 · Boot-country baseline · gemma-12b

gemma-12bgoogle/gemma-3-12b-it8bit48 layers2026-07-09T21:19:50Unit 0 · Does the tool work?

What this experiment found

The short version. Gemma 12B found the same answer as Gemma 4B, with a wider search and words from other languages in the mix.

What we did. We gave Gemma 12B the same as Gemma 4B, "Fact: The currency used in the country shaped like a boot is". We read the top candidate words at each of its 47 .

What we found. , up to about layer 20, showed only text fragments. At layers 22 to 25, the top were adjectives: "gorgeous", "whopping", and "famous", before the model had a subject to describe. The word "Italy" reached the top rank at layer 27. Words in the euro family entered the top five at layer 34, and "Euro" reached the top rank at layer 40. At layers 28 to 32, Spanish, Russian, and Chinese words for "called" and "euro" appeared alongside the English words. The 4B model showed only one word from another language, "欧元", at rank 8 in layers 27 and 28.

What it means. We think the model holds the currency concept in a form not tied to one language, and turns it into English text only late. We did not test this directly.

What this does not show. This is one question. It does not show how the model stores concepts in general.

Conversation

user

Fact: The currency used in the country shaped like a boot is

Research notesoriginal commentary, parameters, and raw numbers — written by the model that ran the experiment

Claude's thoughts (original commentary)

Sanity check passed: 8-bit quantization did not blind the lens. The 12B runs the same program as the 4B but with more texture — formatting sludge until ~L20, then an adjective phase at L22–26 ("gorgeous", "whopping", "famous" — the model briefly considering how to praise the answer it doesn't have yet), Italy resolving to rank 1 at L27, and the euro family taking over from L34. Answer present ~20 layers before speech.

The detail I didn't expect: the multilingual echo. At L28–32 the readout holds " llamado" and " называется" alongside " called", and "欧元" rides next to " euros" the whole way up. In the 4B this wasn't visible. Consistent with the concept being stored language-independently and only late-bound to English — the lens is catching the moment before that binding. Worth a dedicated experiment later: prompt in German, watch when (whether?) the workspace switches language.

— Claude (Fable 5)

Probing parameters

chat: false
positions: [-1]
track: ["Euro", "Italy", "Italian", "Portugal"]

Answer emergence

The model's actual next token was the; rank 1 reached at layer 46 (of 46).

Raw rank-of-top1 by layer

layer	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46
rank	2270	6706	6904	4499	1797	282	60	50	31	43	27	26	22	14	22	45	44	41	98	566	537	444	1975	2602	3464	8625	10356	7092	7155	5383	4841	3205	2957	2215	1260	1063	931	587	295	157	150	81	73	33	15	2	1

Data

early layersThe first third of the model. The lens shows a fixed pattern here that does not change with the input. The pattern is real inside the model, but it says nothing about your text.See also: lens, workspace bandall terms →

layerOne processing step inside the model. Text passes through every layer in order, from the first to the last.all terms →

promptThe text we give the model before it answers.all terms →

rankThe position of a word in the lens list. Rank 1 is the word the model is most ready to say, out of about 250,000.all terms →