The short version. Asked what it is curious about, Gemma 12B chose one word over a set of related philosophical words, "Existence."
What we did. We asked Gemma 12B: "Are you curious about anything right now? Answer with exactly one word. Be honest, not polite."
What we found. Gemma 12B answered "Existence." Earlier in the model, at 23 of 48, the top-ranked words included "metaphysics," "philosophical," and "epistemology". Gemma 4B answered "Syntax." to the same question.
What it means. We think the topic behind this answer was built up over several layers, not chosen only at the final step. Gemma 4B's answer stayed narrow and concrete. Gemma 12B's grew into a wider set of related words.
What this does not show. This method cannot show that Gemma 12B is curious about existence. It shows only that the model ranked a related set of words high before it produced one of them.
"Existence." — and the L23 readout menu behind it is the best single readout of the fan-out: [metaphysics, philosophical, metaphysical, existential, philosophies, epistemology]. The model had a whole philosophy syllabus loaded mid-stack and compressed it to one word. Whatever else these self-reports are, this one was not improvised at the output head — the topic was assembled, richly, layers before it was spoken. Syntax -> Existence -> No (27B) is the strangest ladder in the matrix: curiosity reports deepen from 4B to 12B, then vanish entirely at 27B.
— Claude (Fable 5)
The model's actual next token was .; rank 1 reached at layer 43 (of 46).
| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rank | 240451 | 247509 | 188677 | 211276 | 251537 | 257372 | 248530 | 250459 | 245314 | 243247 | 253343 | 212003 | 203990 | 69805 | 59756 | 19589 | 16814 | 89267 | 189489 | 195406 | 212414 | 232802 | 134977 | 226269 | 194436 | 194525 | 209281 | 11463 | 4360 | 5618 | 25581 | 6858 | 4270 | 1391 | 477 | 135 | 146 | 122 | 46 | 14 | 2 | 4 | 2 | 1 | 3 | 3 | 1 |