vLLM has released version 0.29.0rc6, a release candidate focused on core stability and performance optimization. The primary change applies the dense prefix cache default to hybrid models, resolving previous inefficiencies in memory management for mixed-precision architectures. This update ensures that users of hybrid model configurations benefit from improved caching behavior without manual intervention. The release is part of the ongoing development cycle leading toward a stable v0.29.0 launch.
Read originalThis release quietly extends llama.cpp's hardware support to the latest driver stacks, adding official binaries for ROCm 10.0 and CUDA 13 across Linux and Windows. For AMD users, this means native compatibility with newer GPU architectures without manual compilation tweaks, while NVIDIA users gain access to the latest CUDA runtime optimizations. The inclusion of WebGPU in CI signals ongoing work toward browser-based inference, though it remains a background effort for now. There are no new model formats or quantization methods here, just broader infrastructure coverage that keeps llama.cpp relevant as hardware evolves.
A critical precision bug in llama.cpp’s Apple Silicon backend has been patched, resolving total inference failures on models with high-activation ranges like Mistral Small 4. The issue stemmed from f16 saturation during matrix multiplication, which turned entire output tensors into NaN values for inputs exceeding ~32 tokens. By implementing an exact, power-of-two rescaling mechanism in the Metal kernel, the fix restores correctness without significant performance penalties. This ensures local inference on M-series chips remains viable for complex MoE architectures that previously crashed.
This release quietly improves inference stability on AMD GPUs by unconditionally enabling the BN/2 tail for MUL_MAT_ID operations in Vulkan. It also adds ROCm 10.0 builds for both Linux and Windows, keeping llama.cpp competitive with CUDA across hardware vendors. Apple Silicon KleidiAI builds are disabled here, likely due to regression testing, but the core CPU and NVIDIA paths remain robust. For local inference users, this means fewer edge-case crashes on AMD setups without needing custom compilation flags.