Models & Labs

vLLM v0.31.0: SM100 defaults and fast restarts

vLLM ReleasesOctober 6, 2026high confidence

Why it matters

→SM100 GPUs now get optimized defaults for DeepSeek-V4.1, reducing setup friction for Blackwell users.
→Fast restart capabilities drastically reduce cold-start latency by keeping weights in GPU memory.
→Model Runner V2 stabilizes speculative decoding on large MoE models, preventing common OOM errors.

vLLM has released version 0.31.0, a major update focused on optimizing inference for NVIDIA Blackwell (SM100) architecture and improving operational stability. Key technical changes include setting FlashMLA with NVFP4 compressed KV caches as the default for DeepSeek-V4.1-Flash models, significantly reducing memory footprint. The release introduces a 'fast restart' mechanism using a weight-cache daemon, allowing engines to reload without re-downloading or recomputing quantized weights. Additionally, Model Runner V2 enhances speculative decoding support and fixes memory management issues in Mixture-of-Expert (MoE) deployments. Breaking changes include the removal of slow tokenizer modes and gated multimodal kwargs for improved security.

Read original

The story around this

TopicvLLM Software Updates

Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.

How we got here

1
vLLM Boosts Transformers Backend for Native-Speed Inference
Hugging Face Blog · July 8, 2026 · Related
2
vLLM v0.28.0 Release: Major Performance Enhancements
vLLM Releases · August 28, 2026 · Same story
3
llama.cpp b11037 fixes GPU inference bugs
llama.cpp Releases · September 19, 2026 · Related
4
vLLM v0.30.0: DeepSeek V4.1 and Fast Start
vLLM Releases · September 22, 2026 · Same story
5
llama.cpp optimizes NVIDIA V100 inference performance
llama.cpp Releases · October 2, 2026 · Related

More from vLLM Releases

Coding Toolscoding

vLLM v0.31.0rc3 adds Model Runner V2 dummy inputs

The v0.31.0rc3 release of vLLM brings a critical infrastructure tweak to the new Model Runner V2: support for randomized dummy inputs. This isn't a feature for end-users but a developer-facing fix that stabilizes how the runner handles initial tensor shapes during compilation and warm-up phases. By allowing randomized inputs, it reduces the likelihood of shape-mismatch errors when tracing models with dynamic dimensions. For builders running large-scale inference workloads, this means fewer silent failures and more robust model loading sequences in production environments.

vLLM ReleasesOct 2, 2026

More in Models & Labs

Models & Labsmodels

llama.cpp v0.6.0 adds decision model API and Apple Silicon speedups

This release shifts llama.cpp from a pure text engine to a multimodal inference runtime capable of handling 'decision models' like Clef and GLM-5.3-Flash. The new /v1/systemone server endpoint standardizes how these non-autoregressive models are queried, while the extended batch API allows mixed token and embedding inputs for complex architectures. Apple Silicon users get a tangible performance boost with new Metal MMA kernels that accelerate speculative decoding by up to 3x. It’s a significant step toward supporting the next generation of hybrid reasoning models locally.

llama.cpp ReleasesOct 6, 2026

Models & Labsmodels

Falcon-Emirati-7B targets dialect nuance

Most Arabic models treat the language as a monolith, missing the cultural and linguistic depth of specific dialects. Falcon-Emirati-7B closes this gap by fine-tuning on native Emirati text, synthetic data constrained by strict glossaries, and cultural heritage knowledge. It tops the new Alyah benchmark with 84.83%, proving that scale alone doesn't buy dialect competence. This release underscores a critical shift: true multilingual capability requires targeted adaptation, not just larger parameter counts.

Hugging Face BlogOct 6, 2026

Models & Labsmodels

Reflection debuts Beam open-weight model

Reflection AI is challenging the Chinese dominance in open-weight models with Beam, a 501B-parameter MoE model that claims to match Z.ai’s GLM-5.2 on reasoning benchmarks while using significantly less inference compute. Backed by $4.7 billion and secured GPU deals worth over $7 billion, this two-year-old startup is positioning itself as the Western alternative to DeepSeek and Qwen for enterprise and sovereign AI deployments. The model targets developers and institutions needing cost-effective, localizable infrastructure rather than just raw API access. With weights releasing this month, Beam offers a tangible option for those looking to reduce reliance on closed labs or Chinese open-source ecosystems.

TechCrunch AIOct 5, 2026