Models & Labs
llama.cpp b10984 release enhances CUDA support
llama.cpp Releaseshigh confidence
Why it matters
- →Enhances CUDA support for better performance in AI tasks.
- →Improves compatibility across multiple operating systems and hardware configurations.
- →Strengthens llama.cpp's position as a versatile AI framework.
The b10984 release of llama.cpp focuses on improving CUDA support, particularly for row-contiguous SUM_ROWS operations. This update includes code organization and the addition of GGML_OP_MEAN to handle row-contiguous tensors using a shared kernel. A test for MEAN permute/slice has been added to ensure functionality. The release supports a wide range of platforms, including macOS, Linux, and Windows, with specific enhancements for CUDA and Vulkan environments. This update enhances the framework's compatibility and performance across different systems.
Read originalMore from llama.cpp Releases
Coding Toolscoding
llama.cpp b10993 adds ROCm 10 and CUDA 13 builds
This release quietly extends llama.cpp's hardware support to the latest driver stacks, adding official binaries for ROCm 10.0 and CUDA 13 across Linux and Windows. For AMD users, this means native compatibility with newer GPU architectures without manual compilation tweaks, while NVIDIA users gain access to the latest CUDA runtime optimizations. The inclusion of WebGPU in CI signals ongoing work toward browser-based inference, though it remains a background effort for now. There are no new model formats or quantization methods here, just broader infrastructure coverage that keeps llama.cpp relevant as hardware evolves.
llama.cpp Releases
Coding Toolscoding
llama.cpp fixes Metal NaN bug for large activations
A critical precision bug in llama.cpp’s Apple Silicon backend has been patched, resolving total inference failures on models with high-activation ranges like Mistral Small 4. The issue stemmed from f16 saturation during matrix multiplication, which turned entire output tensors into NaN values for inputs exceeding ~32 tokens. By implementing an exact, power-of-two rescaling mechanism in the Metal kernel, the fix restores correctness without significant performance penalties. This ensures local inference on M-series chips remains viable for complex MoE architectures that previously crashed.
More in Models & Labs
© TechCrunch AIModels & Labsmodels
Salesforce and Nvidia Launch Koa AI Model
Salesforce's launch of Koa, a reasoning model developed with Nvidia's Nemotron, represents a pivotal moment for enterprise AI. Koa is specifically designed for sales and customer support tasks, offering an open-weight model that prioritizes privacy by not utilizing customer data. This approach contrasts with the broader, more theoretical models from frontier labs, focusing instead on practical applications that reduce costs and enhance efficiency. By integrating Koa into its Agentforce platform, Salesforce provides a tailored alternative to models like Claude and ChatGPT, emphasizing secure and efficient AI solutions for businesses.
TechCrunch AI
Models & Labsmodels
