Startups Innovate Beyond Transformers in LLMs

Startups are exploring new methods to enhance large language models (LLMs) beyond the traditional transformer architecture. Subquadratic is developing a sparse attention mechanism to reduce computational load, while Manifest AI is using power retention to manage data more efficiently. Liquid AI combines transformers with liquid neural networks for adaptable and energy-efficient models. These efforts aim to address the limitations of transformers, potentially leading to faster and more efficient LLMs capable of handling larger and more complex data sets.