A new open-source project called TrnRT is designed to run large language models on AWS Trainium2 chips, and its developers say a single trn2.48xlarge server is enough to handle the job. The engine serves an OpenAI-compatible API, which means applications built for OpenAI's client libraries can point at it with minimal changes.
The project, published under the Apache License 2.0 and built by the company RunAnywhere, currently supports one model: GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model with 18 billion active parameters, FP8 precision and a 1-million-token context window. The developers say more open models are on the way. The engine, router, caches and compile pipeline are built to accept additional model families, and the repository includes a document describing what a new model would need.
Much of the LLM optimization work sits in the decode stage, where the model generates output one token at a time. TrnRT runs decode as what the team calls NKI megakernels, fused kernels that each cover roughly half a layer of the model. The fused kernels handle KDA attention, a fused DSA layer, mHC and the mixture-of-experts feed-forward network. The developers write that decode on this hardware is not limited by memory bandwidth. Instead, small instructions, kernel boundaries and exposed collectives dominate the work, so each megakernel keeps its working set in on-chip memory, uses both physical cores symmetrically and moves collective operations inside the kernel.
The engine also includes a prefix cache built as a radix tree over FP8 key-value pages. Evicted pages and state checkpoints can move to a host-memory tier, which the team says helps keep repeated context from being recomputed. Each trn2.48xlarge box runs two engines, and each engine is tensor-parallel across 32 logical NeuronCores. A session-affine router keeps a conversation on the same engine so it can reuse its cached prefix. Developers can pass a session ID through the API to make this happen.
How do you set up TrnRT on AWS Trainium2?
Setup involves two machines. Compiling the model happens offline on any x86_64 host that does not need a NeuronCore, and the README warns that the compile step takes hours. The compiled graphs are then copied to the trn2.48xlarge server, which loads them and, by default, compiles nothing. The Quickstart calls for Docker, the Neuron SDK 2.32 with neuronx-cc 2.27, and Python 3.13. The model weights are downloaded from Hugging Face.
How fast is TrnRT's decode throughput?
The project's benchmarks use the SemiAnalysis AgentX test, which replays real coding-agent sessions. One trn2.48xlarge ran the test at a 1-million-token context with 32 concurrent agents, for one hour per data point. At a decode batch size of 4, the server handled 19.3 requests per minute and produced 46.6 tokens per second per user at the median. At batch 8, it handled 25.2 requests per minute, with 29.4 tokens per second per user. The default batch-12 configuration delivered 28.9 requests per minute and 3.57 million tokens per minute, with a median of 22.3 decode tokens per second per user. Median time to first token with a cached prompt was 6.53 seconds at batch 12, down from 76.73 seconds at batch 4.
What does TrnRT cost per million tokens?
Cost figures depend on how the hardware is rented. The team lists a spot price of $8.60 per hour in us-east-2 as of November 3, 2026, which works out to $0.040 per million tokens at batch 12. On a capacity block priced at $35.76 per hour, the same configuration costs $0.167 per million tokens. The team counts both input, including cached input, and output tokens in those totals.
The developers note that the benchmark numbers came from an earlier build of TrnRT. They say the current repository passes three checks against that build: graph equivalence across all 32 ranks for 157 graphs in the default batch-12 setup, byte-identical outputs on 384 of 384 test requests, and an end-to-end pass of the two-engine serving setup described in the Quickstart.
TrnRT credits the AWS Neuron team for NKI and its documentation, Z.ai for releasing GLM-5.3-Flash, and SemiAnalysis for AgentX and its public traces. The project states that it is independent and is not affiliated with or endorsed by Amazon Web Services.
Este artículo fue escrito con la asistencia de IA.
News Factory APP - noticias agénticas para impulsar tu SEO y AEO.
