Together AI has launched Together Compiler, a system that uses teams of AI agents to rewrite the GPU code behind the open source models it serves. The company says new open models now get faster overnight instead of over several weeks, and customers on its most popular serverless endpoints get the speedup automatically at the same price.
It is a notable bet on agents doing a job that has usually needed scarce, expensive experts. Together AI calls it the first step toward what it describes as recursive self-improving inference, meaning AI that keeps making the systems that run AI faster.
What Together Compiler Actually Does
Every AI model runs on low-level GPU programs called kernels. Tuning those kernels for a new model or a new chip has traditionally taken top researchers days or weeks of hand work. Together Compiler hands that job to a swarm of AI agents instead.
According to the company’s official product page, the process works in a few stages:
- Several agents each start with a different optimization strategy and propose changes to the model’s kernels.
- A persistent test engine hot-reloads each change and checks it for both speed and correctness against the stock engine.
- Only verified changes reach production, and the best strategies are combined.
- Winning kernels, plus the reasoning behind them, go into a shared registry so the next model or chip does not start from scratch.
Together AI says this design runs 252 times more experiments than existing systems that use agents for inference optimization. In older setups, the company notes, validating a single agent’s edit can take up to 15 minutes, which keeps costly GPUs waiting.
Why speed of iteration matters
The company argues that AI traffic is changing too fast for one-time tuning. It says 80% of OpenRouter traffic goes to models released in the last three months, and that the ten most popular open models there each use a different architecture. When the model, chip or workload changes, Together Compiler can simply run the optimization again.
Together Compiler Benchmark Claims
These figures come from Together AI’s own testing and have not been independently confirmed. Still, they give a sense of the gains the company is promising after less than a day of agent work.
| Model | Hardware | Company-reported result |
|---|---|---|
| Kimi K3 | Not specified | 1.5x higher peak throughput than the best InferenceX result |
| GLM 5.3 | Not specified | 1.4x higher peak throughput than the best InferenceX result |
| Llama 1B | Nvidia H100 and B200 | Over 2,100 tokens per second, up to 1.8x hand-written megakernels |
| MiniMax M3.1 Flash | Nvidia B200 | 1,066 tokens per second for one user, 1.5x the baseline |
| DeepSeek V4.1 Flash | Nvidia GB300 | 1,163 tokens per second for one user, 1.2x the baseline |
The official page also claims gains beyond Nvidia hardware. At the same per-user speed, it says GLM on AMD MI355X chips serves 1.2 to 3.7 times more tokens per GPU, and MiniMax M3 on AWS Trainium2 serves 2 to 2.7 times more tokens per chip.
One early customer result
Voice AI startup Phonic, a Together AI customer for over two years, says the compiler cut its median time to first speech by nearly 50%. Co-founder Moin Nadeem said the improvement “happened overnight,” which let the team ship its Phonic-1 model sooner.
Who Can Use It Today
Together Compiler is already running on several of the company’s most-used serverless models, including GLM 5.3, GLM 5.3 Flash and DeepSeek V4.1 Flash. There is no setting to switch on, and Together AI says pricing stays the same.
Customers with dedicated model or container deployments can join a limited beta and ask Together’s research team to optimize their setup. Self-serve access is listed as coming soon, with no firm date given.
Part of a Bigger Push on AI Inference
Together Compiler is the first project in a wider research effort the company calls Polymorphic Inference. The idea is inference that adapts on its own to each deployment’s model, hardware and traffic, rather than a single setup tuned once and left alone.
The team behind it has a strong track record in this area, including the widely used FlashAttention technique and the ThunderKittens kernel library. The launch also lands as the race to cut the cost of AI inference scaling heats up, following big bets like the recent Fal funding round for inference infrastructure.
The takeaway: if agents can reliably tune GPU code in hours, the gap between a new open model’s release and running it cheaply could shrink dramatically.
What This Means for You
If you build apps on open models, this kind of tooling could quietly lower your costs and speed up responses. Here is what to keep in mind:
- If you already use Together AI serverless endpoints for supported models, you may see faster responses without changing any code.
- Treat the benchmark numbers as vendor claims until outside tests confirm them, and measure latency on your own workloads.
- Faster tuning makes it easier to switch to newer open models soon after release, which matters if you are weighing AI developer tools and model providers.
- Agent-driven code changes still need strong checks, so Together’s correctness testing step is worth asking about if you plan to use the beta.
For more on how agents are spreading into everyday computing, see our look at local AI agents, or browse the latest AI news updates.
Sources
Primary source: Together AI’s Together Compiler announcement. Additional details from the company’s press release via Yahoo Finance.
Conclusion
Together Compiler puts AI agents in charge of one of the hardest jobs in AI infrastructure: squeezing more speed out of GPUs for every new model. Together AI says it can now tune open models in under a day, with gains of up to 3x on some setups, and it is already live on popular serverless endpoints at no extra cost. The numbers are the company’s own, so independent testing will matter. If they hold up, faster and cheaper access to new open models could reach developers much sooner than before.
