知Yizhi Daily
← Back to this edition

Conversations

RunAnywhere Open-Sources TrnRT, an LLM Inference Engine for AWS Trainium

Edition 2026-11-18 · 08:00 Beijing time

RunAnywhere posted the open-source project TrnRT, or Trainium RunTime, on Hacker News. It is a large-model inference engine aimed at AWS Trainium. The project is three weeks old and can currently run the FP8 version of GLM-5.3 flash on trn2.48xlarge, with community contributions welcome.

Poster Sanchit says a single device can serve two GLM-5.3 flash instances through a cache-aware router using eight trn2 units; on the SemiAnalysis AgentX benchmark, it can handle 3.57 million TPM. AgentX replays real programming-agent sessions, with results including 28.9 RPM at batch 12 and 32 agents; with 16 agents and prompt caching, P50 TTFT is about 1.24 seconds and decoding speed is 35.9 TPS.

The team chose Trn2 because it wanted to explore its cost efficiency relative to H100/H200. The poster says large MoE models such as GLM-5.3 flash, which has an approximately 1,000,000-token context window, require large amounts of memory, while the memory cost per terabyte on trn2.48xlarge is less than half that of eight H200 GPUs on AWS. Because existing vLLM Neuron does not register the GLM model, NxDI is no longer under active development, and SGLang Neuron also has insufficient Trn2 support, the team wrote its own NKI kernels: each layer uses one kernel for half-side decoding, and the 11 sparse-attention layers fuse two blocks; a mega kernel lets an entire layer require only two dispatches, speeding up batch-8 decoding by 25%. The poster also notes that sparse-attention prefill involves approximation, affecting about 1.2% of block selection.

Original sources

Hacker News ↗
Back to this editionOpen interactive reader