vLLM has released version 0.25.0, introducing Model Runner V2 as the default for all dense models, enhancing execution with features like real-time embeddings and dynamic speculative decoding. The update also removes the legacy PagedAttention, streamlining the platform's performance. New models such as LLaVA-OneVision-2 and Unlimited OCR have been added, expanding the model zoo. These changes aim to improve efficiency and broaden the capabilities available to developers using vLLM.
Read originalThe latest b9974 release of llama.cpp addresses a critical issue for CUDA users by preventing crashes when querying memory on devices with no free memory. Previously, attempting to check memory availability could lead to a fatal crash if the device was out of memory. The update now assigns zero total/free memory to such devices, ensuring the fit algorithm doesn't attempt to use them, thus avoiding crashes. This change enhances stability for CUDA-enabled builds, especially when users specify '-dev none'. While the update doesn't introduce new features, it significantly improves reliability for developers working with CUDA devices.
The latest b9977 release of llama.cpp addresses a critical issue in the conversion process between Anthropic and OpenAI formats, where image blocks in tool results were being dropped. This fix ensures that multimodal tool outputs, such as those returning images, are correctly processed and received by the model. By converting image blocks into OpenAI's multimodal content parts, the update maintains backward compatibility with plain-text results. This release is a technical refinement that enhances the robustness of multimodal AI applications, ensuring seamless integration across different platforms.
The latest b9980 release of llama.cpp introduces a significant change in how multimodal models are handled. By ensuring that models with the --no-mmproj-auto flag are not mistakenly identified as supporting image or audio inputs, the update clarifies the capabilities of models on the /v1/models endpoint. This release also includes a variety of platform-specific builds, from macOS and Linux to Windows and openEuler, enhancing compatibility across different environments. While the update doesn't introduce new model architectures, it refines the existing framework, making it more reliable for developers working with multimodal AI applications.