What is Respan's open-source AI gateway?
Respan has released an open-source AI gateway written in Rust, and the company is betting that developers want to build their own features on top of it rather than wait for someone else to add them. The gateway is designed around plugins. Almost everything, including model providers, communication protocols and features such as fallbacks and personal data redaction, is a plugin that sits outside a small core.
The launch was posted to Hacker News by Andy, a member of the Respan team. The company, which goes by Respan and is part of the Y Combinator W24 batch, says the design grew out of problems it ran into while running a gateway at scale.
Respan has been building its gateway on top of LiteLLM since 2024. The gateway now serves thousands of organizations, and each one needs its own API key settings, routes and limits. To handle that, the team kept wrapping LiteLLM and adding features. At one point, the wrapper code grew past 100,000 lines.
Why did Respan redesign its LiteLLM-based gateway?
The breaking point came this January. A routine LiteLLM upgrade caused intermittent connection errors, and the team had to roll it back. Respan says it still has not determined the cause. Since then, it no longer upgrades LiteLLM directly. Instead, engineers manually backport upstream fixes. The team concluded that continuing to patch LiteLLM was not a workable long-term approach and decided to redesign the gateway itself.
The new design takes its cue from Nginx. Respan says an AI gateway should have a very small core that does one job: dispatching events. Everything else is a plugin, including providers such as OpenAI, Anthropic and Gemini, protocols such as chat, realtime voice and MCP, and modalities such as images and videos. Gateway features for reliability and security, including smart routing, fallbacks and PII redaction, are also plugins, and the company has published example versions of several of them.
Respan says adding a provider is intentionally simple. A Mistral provider, for example, is 22 lines long. Quirks in how a provider behaves are handled the same way. The company gives the example of Groq placing token usage in its own field, which gets a dedicated plugin and its own test.
The gateway also lets applications mix and match APIs. Respan says an application built on the OpenAI Realtime API can use Gemini Live through the gateway, although not every feature translates across providers. A simple chat app can use MCP tools configured in the gateway without implementing MCP itself. The company also says that when a client and provider use the same API, the gateway keeps provider-specific fields in their native format, even when its shared request structure has no field for them, so those fields are not dropped.
How do plugins work in the Respan gateway?
Under the hood, an adapter passes each HTTP or WebSocket connection to the first client format that recognizes it. That format exposes the request as a two-way stream of typed input and output events. The core runs plugins in the order they were registered. Those plugins can choose a provider, filter or wrap the stream, make additional calls, return a response or refuse the request. If the client and provider use the same format and no plugin changes the payload, the original bytes pass through unchanged. When formats differ, the gateway translates through shared types for messages, tool calls, results, streams and usage.
The code is available on GitHub under the MIT or Apache-2.0 license. Developers do not need Respan's hosted platform to use it. They can build the gateway from source, add their own provider keys and point a supported client at the gateway by changing its base URL. Respan describes the project as pre-alpha.
Respan makes money from its hosted platform, which runs the gateway for customers and adds observability, evaluations and unified billing. The company is asking outside developers to contribute plugins for routing, tools, filtering, observability and security for AI agents. A file called PLUGINS.md in the repository explains how to write one and links to the plugins Respan has built so far. The company says building out that ecosystem is central to its roadmap.
Este artículo fue escrito con la asistencia de IA.
News Factory APP - noticias agénticas para impulsar tu SEO y AEO.