Models & Labs
Kog Optimizes GPUs for Faster AI Inference
TechCrunch AIhigh confidence
Why it matters
- →Kog's approach could drastically reduce AI inference times without new hardware investment.
- →This innovation may provide a cost-effective solution for enterprises relying on AI workflows.
- →Successful scaling to larger models could redefine GPU capabilities in AI applications.
French startup Kog is pushing the boundaries of AI inference by optimizing existing GPUs for faster performance. Their recent tech preview showcased the ability to achieve rapid single-request decoding on standard datacenter GPUs, such as AMD MI300X and Nvidia H200. This approach could significantly reduce inference times, addressing a critical bottleneck in AI workflows. While Kog faces the challenge of scaling its solution to larger models, its focus on software optimization rather than new hardware could offer a cost-effective alternative for enterprises. The startup aims to demonstrate its approach on large language models by September to secure further funding.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
More from TechCrunch AI
More in Models & Labs
Models & Labsmodels
llama.cpp b11535 release with ROCm 10 and CUDA 13
This release quietly cements llama.cpp as the universal inference runtime by adding default support for ROCm 10.0 and CUDA 13.4 across Linux and Windows. AMD GPU users finally get parity with NVIDIA's latest driver stack without manual configuration, while Apple Silicon KleidiAI builds are temporarily disabled to resolve stability issues. The inclusion of Snapdragon NPU support on Linux signals a serious push into edge AI hardware beyond just x86 and ARM CPUs. It is less about new features and more about ensuring the toolchain keeps pace with the rapidly evolving GPU landscape.
llama.cpp Releases
Models & Labscoding
llama.cpp b11540 adds ROCm 10 and CUDA 13 support
This release quietly closes the hardware gap for local inference by adding default builds for ROCm 10.0 and CUDA 13.4 across Linux and Windows. AMD users finally get parity with NVIDIA in the binary distribution, while CUDA 13 support future-proofs setups on newer drivers. The inclusion of Snapdragon and OpenVINO binaries further broadens the hardware surface area without requiring custom compilation. It is a pragmatic update that makes llama.cpp the most accessible runtime for diverse local AI hardware.
