Every conversation with a language model leaves a trail of internal states, and a researcher has now built an interactive atlas to see what those trails look like. The project maps thousands of conversations inside Qwen3-1.7B, an open model, and finds that they form thin, overlapping sheets in a three-dimensional view. The method is exploratory, and the author says the map leaves out most of the model's variation.
How does a language model represent text internally?
The starting point is the model's embedding table. Text is broken into tokens, which can be whole words, parts of words or punctuation. Each token selects a row from that table, a learned list of numbers that gives the model its initial representation. The layers then read from and add to this shared workspace, known as the residual stream, as they process the text. In Qwen3-1.7B, each position's vector contains 2,048 numbers.
The author began with 900 English question-and-answer examples across 100 topics. The model's answers were recorded and processed through the same frozen model. When plotted, the English token states appeared to occupy a thin sheet. A best-fitting plane captured about 99.3 percent of the variation within the three-dimensional projection.
Does the language of a conversation change how the model groups it?
That result looked like a sign of compression, but the author wanted to test something more basic than topic. Chinese versions of the same topics and question intents produced a separate cluster, visibly offset from English. The answers were generated independently rather than translated, so the separation could reflect language, writing system, tokenization, wording or answer quality. The author said the geometry alone could not separate those causes.
German and Croatian, both Latin-script languages, overlapped substantially with English. Chinese stayed more distinct in the same view. The picture showed regions with different centers and shapes, not sealed compartments. The author also cautioned that the size of a patch is not a direct measure of capability. Croatian responses contained many more Qwen tokens than the English responses, which produces more points and more line segments.
Code was the next test. The 90 Python examples covered list processing, sorting, binary search, graph traversal, JSON, dates, matrices, recursion and classes. Their prompts were in English, and their answers were independently generated Python. The coding paths appeared in the same broad neighborhood as English and the other Latin-script datasets. A simple distance comparison, however, put the coding set closer to Croatian and German than to English. The author noted that being visually nearby and being numerically closest are different claims.
The mapping method also changed the picture. The standard Global PCA frame was fit using vocabulary embeddings and conversation averages, and vocabulary embeddings made up roughly 98 percent of the fitting points. The author then tried a Conversation PCA frame, which selects directions using only the conversation averages. Under that approach, the three directions captured 80.1 percent of the variation between conversation averages, compared with 42.4 percent for the three Global PCA directions. German and Croatian separated into more distinguishable regions, though overlap remained.
What does the interactive map leave out?
The limits are significant. The author checked the flatness claim against all 264,571 final token states in their original 2,048-dimensional space. The visible sheet held almost all the variation in the three-dimensional view, yet the displayed Global PCA axes captured only 17.6 percent of the full token-state variation. The best possible two-dimensional plane in the full space captured 28.7 percent. Retaining 90 percent of the full variation required 684 directions.
The author described the results as associations within this dataset and projection, not established findings about the model in general. The sample was small and deliberately constructed, and the animations only rotate the camera. They do not simulate how the model computes an answer. The atlas is best read as a way to ask better questions about multilingual representations, the author suggested, rather than as proof of a shared structure across all languages and tasks.
Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.