Models & Labs
OpenVINO Backend Enhancements in llama.cpp b10419
llama.cpp Releaseshigh confidence
Why it matters
- →Enhancements improve memory efficiency, crucial for large model deployments.
- →Fixes to mixed-rank broadcasts enhance accuracy in GPU inference.
- →Memory optimizations reduce resource usage without affecting performance.
The b10419 release of llama.cpp introduces several enhancements to the OpenVINO backend, aimed at improving memory management and operational accuracy. Key updates include fixing mixed-rank broadcast issues in GPU plugins, which significantly improved perplexity scores. Additionally, a new mode reduces memory usage by releasing host weight buffers after model compilation, effectively halving the steady-state RSS. These updates enhance the efficiency and reliability of the OpenVINO backend, particularly for GPU-based inference tasks.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
How we got here
- 1llama.cpp b9817 release enhances OpenVINO support
llama.cpp Releases · June 28, 2026 · Same story
What happened next
- 2Llama.cpp b10981 release enhances OpenVINO support
llama.cpp Releases · September 16, 2026 · Same story
More from llama.cpp Releases
Coding Toolscoding
llama.cpp b11372 optimizes Qwen4-Exp memory and GPU support
This release tackles the notorious memory hunger of long-context inference for Qwen4-exp models by halving indexer score memory. The optimization works by computing head scores in place rather than materializing separate tensors, a change that significantly reduces VRAM pressure during heavy workloads. Beyond memory efficiency, b11372 expands hardware coverage with CUDA 13 support and Vulkan tiling for the lightning indexer. It also adds ROCm 10.0 binaries, keeping AMD users in step with NVIDIA's latest driver ecosystem. The result is a leaner runtime that handles extended contexts without hitting out-of-memory errors as quickly.
llama.cpp Releases
Coding Toolscoding
llama.cpp b11374 optimizes OpenVINO MoE performance
This release significantly tightens llama.cpp’s integration with Intel’s OpenVINO backend, specifically targeting Mixture of Experts (MoE) models like Qwen3.5 and Gemma-4. By fusing MoE routing and GDN normalization operations, prefill throughput on Arc GPUs jumps from 66 to over 1,600 tokens per second, effectively removing a major bottleneck for local inference on Intel hardware. The update also fixes critical stateful execution bugs that previously caused crashes or incorrect axis handling during decoding. This makes OpenVINO a far more viable option for running complex MoE architectures on consumer-grade Intel GPUs without relying on NVIDIA CUDA.
