Models & Labs
Together AI Introduces Autoscaling for LLM Inference
Together AI Bloghigh confidence
Why it matters
- →Autoscaling based on inference-native metrics optimizes resource use for LLM deployments.
- →It addresses the limitations of traditional autoscaling methods in handling LLM traffic.
- →This feature is crucial for managing costs and performance in GPU-constrained environments.
Together AI has launched a new autoscaling feature for large language model (LLM) inference on its platform. This feature allows deployments to scale based on specific metrics such as in-flight requests and GPU utilization, addressing the unique challenges of LLM serving. Traditional autoscaling methods often fail due to misleading metrics and lengthy cold starts, but Together AI's approach offers a catalog of inference-native metrics for more precise scaling. This advancement helps optimize resource allocation, especially in environments where GPU resources are limited.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
More in Models & Labs
Models & Labsother
llama.cpp b11193 adds Snapdragon Hexagon and ROCm 10
This release quietly expands llama.cpp's hardware support to include Qualcomm's Hexagon NPU on Linux arm64, a significant step for local inference on Snapdragon devices. It also updates CUDA builds to version 13.4 and introduces ROCm 10.0 binaries, keeping the project aligned with the latest NVIDIA and AMD driver ecosystems. KleidiAI on Apple Silicon is temporarily disabled in this build, likely due to stability checks rather than a feature rollback. For developers targeting edge AI or diverse GPU stacks, this update ensures broader compatibility without requiring custom compilation.
llama.cpp Releases
© Lev SelectorModels & Labsmodels
NVIDIA NVFP4 and SoL-Pi Token Efficiency
NVIDIA introduced the NVFP4 4-bit format and SoL-Pi technology, which uses 2x fewer tokens for improved efficiency.