Models & Labs
vLLM v0.30.0: DeepSeek V4.1 and Fast Start
vLLM Releaseshigh confidence
Why it matters
- →DeepSeek-V4.1-Flash support allows MXFP8 KV caching on SM100, significantly reducing memory footprint for large reasoning models.
- →Fast Start daemon enables near-instant engine restarts by caching weights in GPU memory via CUDA IPC, eliminating disk I/O bottlenecks.
- →HiSparse host-resident tier spills KV pages to pinned host memory, extending effective context windows without additional GPU hardware.
vLLM has released version 0.30.0, featuring support for DeepSeek-V4.1-Flash and GLM-5.3-Flash. Key performance improvements include Fast Start, a persistent weight-cache daemon that reduces engine initialization time by mapping weights via CUDA IPC, and HiSparse, which offloads sparse MLA KV pages to host memory under GPU pressure. The release also introduces Gumbel-max watermarking for AI detection and updates the Model Runner V2 with dual-batch overlap in eager mode. Breaking changes include making scale-out endpoints opt-in and removing deprecated environment variables.
Read originalMore in Models & Labs
Models & Labsother
llama.cpp b11080 release with ROCm 10 and CUDA 13
This update quietly extends llama.cpp’s hardware support to ROCm 10.0 and CUDA 13.4 across Linux and Windows, keeping the library competitive as NVIDIA pushes newer driver stacks. The test suite also gains regex filtering for backend operations, a practical improvement for developers debugging specific inference paths. While no new model architectures are introduced, this release ensures compatibility with the latest GPU ecosystems without forcing users to wait for major version bumps.
llama.cpp Releases
Models & Labsother
llama.cpp b11081 release with CUDA 13 and ROCm 10 support
This release quietly cements llama.cpp as the universal inference runtime by adding default builds for CUDA 13.4 and ROCm 10.0, effectively closing the gap on newer NVIDIA and AMD hardware without requiring manual compilation flags. The inclusion of KleidiAI for Apple Silicon remains a key differentiator for local Mac users seeking optimized ARM kernels. While the changelog details internal test improvements like configurable tensor standard deviation, the real value lies in the expanded binary matrix that supports developers across the latest GPU architectures. Readers can now deploy on cutting-edge hardware out of the box. The update ensures that teams using the newest graphics cards do not need to wait for source code patches to achieve stable performance. This is a practical step toward making local inference truly hardware-agnostic.
llama.cpp Releases
© Duncan RogoffModels & Labsother
Jev AI Classifier: Speed vs. Hype
Jev positions itself as a specialized classifier claiming massive speed and cost advantages over frontier models like Opus and Fable. The reality is nuanced: it excels at rigid schema-based routing tasks but fails completely on open-ended generation or complex reasoning. This isn't a replacement for general-purpose LLMs, but rather a high-throughput router that can drastically cut inference costs when used correctly. The verdict suggests it's a useful tool for specific architectural patterns, not a magic bullet for all AI workloads.
Duncan Rogoff