A software developer is urging people who use large language models to stop relying solely on the tools their AI providers hand them and instead run their own proxy server, hosted on infrastructure they control. The argument appears in a post that questions how anyone can gain LLM visibility into what is happening with their AI usage without one.
The author starts with the typical setup. Most users chat through a frontend tied to a paid subscription, whether that is a terminal interface, a graphical app or a voice tool. Their data goes in and the provider's data comes back. The author says providers present usage screens as proof that work is happening, but those screens are a distraction. They are not the token usage statistics, with time series data, that engineers need.
The post's examples of what people are building range widely. The author mentions someone automating the cloning of new toy releases by turning 2D product images into 3D models, and another person using a model to look up macronutrient information for a fast-food sandwich.
Why run your own LLM proxy instead of provider tools?
The author acknowledges that some providers offer OpenTelemetry export options, then argues that those options do not solve the problem. The author lists several reasons to run a proxy instead. A single endpoint can route across multiple providers and models. Logs can show which key called which model, from which client, at what time and with what status. Virtual API keys can keep a single main key from being scattered across projects, agents and harnesses. And users keep control of their infrastructure, which makes it faster to switch models or providers without reconfiguring everything.
The author does not name a proxy product to use. The advice is to run one somewhere you control and then adapt it. The author says they have modified their own proxy to offer better insight, to support custom MCP server endpoints and to add other features.
What does an LLM proxy reveal during a coding session?
As an example of the payoff, the author says that when a cache miss shows up during a coding session, they can work out why in about 20 seconds by hand. The author has not automated that process yet, but says the proxy has an API, so automation is possible.
The author frames the proxy as a matter of ownership: infrastructure, application visibility, data pathing and storage. The author runs one at home because it is fun and a good way to learn, and at work because, in the author's view, the job is not being done properly without it. The author admits not fully understanding how a large language model works well enough to explain it on a whiteboard in a job interview. The author says the goal is more modest: to understand the system well enough to monitor it while it serves inference.
Cet article a été rédigé avec l'assistance de l'IA.
News Factory APP - actualités agentiques pour booster votre SEO et AEO.