Models & Labs
Together AI automates inference optimization with agents
Together AI Bloghigh confidence
Why it matters
- →Autonomous agents can now optimize inference performance faster than human experts or traditional compilers.
- →Persistent engine architecture drastically reduces overhead, enabling high-throughput experimentation loops.
- →Performance gains generalize across diverse model architectures and hardware platforms without manual tuning.
Together AI has released Together Compiler (TKCC), an autonomous system that uses AI agents to continuously optimize inference workloads for varying models and hardware configurations. The system achieves up to 252x higher experimentation throughput by using persistent engine instances rather than restarting inference engines for each test, enabling rapid iteration over optimization strategies. TKCC has set new state-of-the-art results on the SemiAnalysis InferenceX benchmark, achieving 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash, while outperforming hand-tuned megakernels by 1.7x on Llama 1B. The approach also demonstrates generalizability to non-language domains including weather modeling and partial differential equations.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
How we got here
- 1Intel Explores Agentic AI for Enterprise Workflows
MIT Technology Review AI · July 27, 2026 · Related
- 2AI Agents Transform Scientific Computing
OpenAI · July 28, 2026 · Related
- 3Together AI Enhances Model Inference Configuration
Together AI Blog · July 29, 2026 · Same story
- 4Together AI Introduces Autoscaling for LLM Inference
Together AI Blog · July 31, 2026 · Same story
- 5Hugging Face Introduces ALTK-Evolve for Efficient AI Agents
Hugging Face Blog · August 11, 2026 · Related
- 6Open Source AI Software Factory Automates Coding Agents
Cole Medin · September 12, 2026 · Related
- 7Jev: A Decision-Only AI Model for Agents
Cole Medin · September 21, 2026 · Related
- 8Liquid AI releases efficient on-device agent model
Hugging Face Blog · October 13, 2026 · Related
Follow this story
Open the full story →Together AI automates inference optimization with agents
2 developments
- Nov 4 · Together AI Blog
Together AI automates inference optimization with agents (This article) - Nov 4 · Together AI Blog
Together AI introduces Polymorphic Inference and Compiler
More in Models & Labs
Models & Labsother
llama.cpp b12177 adds CUDA 13 and ROCm 10
This release quietly closes the hardware gap for local inference by adding default support for CUDA 13.5 and ROCm 10.0 alongside existing CUDA 12 builds. AMD GPU users finally get first-class parity with NVIDIA, while Snapdragon mobile chips gain dedicated NPU acceleration paths. Apple Silicon KleidiAI is explicitly disabled here, a notable regression for Mac performance that suggests ongoing tuning issues. The inclusion of OpenVINO and SYCL backends further cements llama.cpp as the universal runtime for inference across nearly every major silicon architecture.
llama.cpp Releases
Models & Labsmodels
llama.cpp adds Kimi K3.1-Pro support
Kimi’s new K3.1-Pro model joins the llama.cpp ecosystem, bringing a distinct architectural twist to local inference. By replacing full-attention MLA layers with DSA (Dynamic Sparse Attention) and linear-attention KDA layers, this release tests how well sparse caching handles hybrid attention patterns. The implementation keeps indexer precision high to avoid quality loss during top-k selection, ensuring the model runs accurately on consumer hardware. This expands the library’s reach into newer Chinese open-weight models that rely on complex memory management.
llama.cpp Releases
Models & Labsother
llama.cpp b12182 release with CUDA 13 and ROCm 10
This release quietly cements llama.cpp as the universal inference runtime by adding default builds for CUDA 13.5 and ROCm 10.0 across Linux and Windows. The inclusion of ROCm 10 means AMD GPU users no longer need custom compilation to access modern acceleration, effectively closing the gap with NVIDIA's ecosystem support. While KleidiAI on Apple Silicon is disabled in this build, the broader platform coverage ensures that local inference remains accessible on virtually any hardware configuration without manual setup.
llama.cpp Releases