Models & Labs
llama.cpp b9775 Release Expands Platform Support
llama.cpp Releaseshigh confidence
Why it matters
- →Expands platform support, enhancing deployment flexibility for developers.
- →Introduces ROCm 7.2 support, benefiting AMD GPU users.
- →Reinforces llama.cpp's versatility in AI inference across multiple environments.
The latest b9775 release of llama.cpp focuses on expanding platform support rather than introducing new features. It includes builds for macOS, Linux, Windows, and openEuler, with notable support for ROCm 7.2 on Ubuntu x64, enhancing options for AMD GPU users. This release ensures that developers can deploy llama.cpp across a wide range of systems, reinforcing its versatility. While not groundbreaking, the update strengthens llama.cpp's role in AI inference across diverse platforms.
Read originalMore from llama.cpp Releases
Models & Labsmodels
llama.cpp b9767 Release Enhances MTP Inference
The b9767 release of llama.cpp introduces significant improvements to MTP inference by optimizing the mat-vec path for small batches, which enhances decoding efficiency. A new barrier in the NUM_COLS loop of the mul-mat-vec process is expected to boost performance. While no new model architectures are included, this update refines the platform's capabilities across macOS, Linux, and Windows. Notably, it supports macOS Apple Silicon, Ubuntu with ROCm 7.2, and Windows with CUDA 12 and 13. This release continues llama.cpp's focus on performance optimization and compatibility, making it a more powerful tool for developers.
llama.cpp Releases
Models & Labsmodels
Granite Speech Plus Support Added in b9768 Release
The b9768 release of llama.cpp expands its capabilities by integrating Granite Speech Plus, which enhances audio processing with multi-layer concatenation. This update is particularly relevant for developers focused on audio applications, as it resolves naming inconsistencies and standardizes feature layer usage. While no new models are introduced, the release fortifies the existing framework, making it more reliable for audio tasks. This iteration marks a refinement in the tool's functionality, especially for those utilizing its audio features.
llama.cpp Releases
Open Sourcemodels
llama.cpp b9771 Release Trims Shader Variants
The b9771 release of llama.cpp brings a notable optimization by setting 'mul_mm ALIGNED' as a spec constant, effectively reducing the shader variant explosion and cutting down the binary size. This change is particularly advantageous for developers using Vulkan, as it simplifies the compilation process. While the update doesn't introduce new features, it continues to enhance the platform's compatibility across macOS, Linux, Windows, and openEuler. This release is a step forward in making llama.cpp more efficient and accessible for developers working with different hardware setups, including Apple Silicon, ROCm, and CUDA environments.
llama.cpp Releases
More in Models & Labs
© NVIDIA BlogModels & Labsagents
NVIDIA Launches Agent Toolkit for Specialized AI
NVIDIA's new Agent Toolkit is a significant step towards creating specialized AI agents that can be customized and trusted by enterprises. By providing a modular foundation of models, tools, and secure runtime, the toolkit allows businesses to build AI systems tailored to their specific workflows. This development is particularly impactful in industries like life sciences and healthcare, where AI agents can drastically reduce the time needed for complex tasks such as protein design and clinical documentation. The toolkit's open nature ensures that companies can integrate these agents into existing systems, enhancing efficiency and control.
NVIDIA Blog
© The Rundown AIModels & Labsmodels
Sakana AI launches Fugu for model orchestration
Sakana AI's Fugu model introduces a novel approach to AI usage by coordinating multiple models through a single API, addressing challenges like those posed by export controls on Anthropic's models. Fugu is available in two versions: a faster model for everyday tasks and a more robust version for complex applications such as patent research. While Sakana asserts that Fugu performs comparably to leading models, initial feedback suggests it may not yet achieve those standards. This launch represents a shift towards model orchestration, though questions about cost and transparency remain unresolved.
The Rundown AI
© Hugging Face BlogModels & Labscoding
Cross-Origin Storage API in Transformers.js
The proposed Cross-Origin Storage API could revolutionize how web apps handle large files across different origins by using cryptographic hashes instead of URLs for identification. This approach aims to eliminate redundant downloads and storage, which is currently a challenge due to browser cache isolation by origin. By allowing shared resources like AI models and Wasm files to be recognized across different apps, this API could significantly reduce bandwidth and storage usage. Although still in early stages and not natively supported by browsers, developers can experiment with it using a polyfill extension.
Hugging Face Blog