A new visualization tool is providing a fascinating glimpse into the inner workings of large language models. By allowing users to hover over generated tokens and see which previous tokens influenced their creation, the tool is helping to demystify the decision-making process of these complex models.
The tool, which uses a React app and Transformers.js, calculates the attention weight of each previous token, scaled by the magnitude of the value vector, aggregated across all attention heads, and summed across all layers. This information is then used to control the opacity of the previous tokens, creating a visual representation of the model's thought process.
One of the most interesting aspects of the tool is its ability to show how large language models are able to copy and paste text with remarkable accuracy. By examining the attention weights, users can see that the model is not simply predicting the next token in a sequence, but is instead drawing upon a wide range of previous tokens to inform its decisions.
For example, in the default "Office Move Summary" prompt, hovering over the text that is copied verbatim, such as addresses and dates, reveals that the generated token takes up a lot of information from the source data. This suggests that the model is able to combine information from multiple sources to create a coherent and accurate output.
The tool also provides insight into the model's ability to combine information from different phrases. In the "Office Move Summary" prompt, hovering over the word "remain" in the phrase "Existing access cards and phone numbers remain" reveals that the model is drawing upon information from the words "work" in "Existing employee access cards will work" and "stay the same" in "company phone numbers will stay the same".
The implementation of the tool is a notable achievement, given the complexity of the models involved. The developer had to modify the onnx file to expose internal values, and then upload a separate instrumented model to a Hugging Face repo to enable browser-based generation.
The code for the tool is available on GitHub, and is a valuable resource for anyone looking to understand the inner workings of large language models. As the use of these models continues to grow, tools like this will become increasingly important for helping us to understand and optimize their performance.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.