Models & Labs
llama.cpp b9974 Release Fixes CUDA Memory Crash
llama.cpp Releaseshigh confidence
Why it matters
- →Prevents crashes on CUDA devices with no free memory, improving stability.
- →Ensures the fit algorithm doesn't attempt to use unsuitable devices.
- →Enhances reliability for developers using CUDA-enabled builds.
The b9974 release of llama.cpp resolves a significant issue for CUDA users by preventing crashes when querying memory on devices with no available memory. Previously, the call to cudaMemGetInfo() could cause a fatal crash if the device was out of memory. The update now assigns zero total/free memory to such devices, preventing the fit algorithm from attempting to use them and avoiding crashes. This update enhances stability for CUDA-enabled builds, particularly when users specify '-dev none'.
Read originalMore from llama.cpp Releases
Open Sourcemodels
llama.cpp b9973 Release Expands Platform Support
The b9973 release of llama.cpp focuses on enhancing compatibility across a wide array of systems, though it doesn't introduce major new features. This update is particularly notable for adding ROCm 7.2 support on Ubuntu x64, offering AMD GPU users a viable alternative to NVIDIA's CUDA. The release continues to provide extensive builds for macOS, Linux, Windows, and openEuler, ensuring that developers can deploy llama.cpp in varied environments. While the update lacks groundbreaking innovations, it strengthens llama.cpp's role as a versatile tool for AI inference, accommodating diverse hardware configurations.
llama.cpp Releases
Open Sourcemodels
llama.cpp b9975 Release Expands Platform Support
The b9975 release of llama.cpp continues its focus on enhancing platform compatibility, though it doesn't introduce major new features. This update includes ROCm 7.2 support for Ubuntu x64, which is a significant development for AMD GPU users looking for alternatives to NVIDIA's CUDA. Although KleidiAI support for macOS Apple Silicon is currently disabled, the release still supports numerous operating systems, including Windows and openEuler. By covering diverse hardware configurations, llama.cpp strengthens its role as a flexible inference runtime, even without new model architectures.
llama.cpp Releases
Models & Labsmodels
Llama.cpp b9977 release fixes image block conversion
The latest b9977 release of llama.cpp addresses a critical issue in the conversion process between Anthropic and OpenAI formats, where image blocks in tool results were being dropped. This fix ensures that multimodal tool outputs, such as those returning images, are correctly processed and received by the model. By converting image blocks into OpenAI's multimodal content parts, the update maintains backward compatibility with plain-text results. This release is a technical refinement that enhances the robustness of multimodal AI applications, ensuring seamless integration across different platforms.
llama.cpp Releases
More in Models & Labs
Models & Labsmodels
vLLM v0.25.0 Release Enhances Model Execution
The vLLM v0.25.0 release marks a significant step forward in model execution and performance. With Model Runner V2 now the default for dense models, users benefit from enhanced support for real-time embeddings and dynamic speculative decoding. The removal of PagedAttention and improvements to the Transformers backend, including FP8 MoE support, streamline operations and boost speed. New models like LLaVA-OneVision-2 and Unlimited OCR expand the model zoo, offering more options for developers. This release solidifies vLLM's position as a robust platform for AI model deployment, with improved efficiency and expanded capabilities.
vLLM Releases
© The Verge AIModels & Labsmodels
Apple's AI Chips Emerge from Self-Driving Car Effort
Apple's self-driving car project may have stalled, but it left a significant legacy in the form of powerful AI chips. The development of the Neural Engine, which debuted with the iPhone X, was a direct result of the need for robust on-device AI processing for the car. This technology has since become a cornerstone of Apple's hardware strategy, enabling advanced features like FaceID and augmented reality while maintaining privacy by reducing cloud data reliance. Looking ahead, Apple is focusing on its M7 Ultra chip, expected to support up to 1.5TB of RAM, marking a significant leap in AI hardware capabilities.
The Verge AI
© Sam WitteveenModels & Labsmodels
Cactus Needle: 26M Parameter Model Outperforms Larger Rivals
Cactus has launched Needle, a function calling model with 26 million parameters that is making a significant impact by outperforming larger models. Its innovative architecture defies the conventional belief that bigger models are always better, demonstrating that efficiency can coexist with high performance. Available on platforms like Hugging Face, Needle invites developers to explore its potential and experiment with its capabilities. This development marks a shift towards designing AI models that prioritize compactness and efficiency without compromising on results.
Sam Witteveen