Startups pursue alternatives to transformer-based LLMs amid efficiency and capability limits

A growing number of researchers and engineers argue that transformer-based large language models (LLMs) are approaching practical limits due to inefficiencies in dense attention mechanisms, which scale poorly with input length and drive up compute and energy costs. OpenAI’s president, Greg Brockman, projected the company would spend $50 billion on computing in 2026, while the International Energy Agency estimated global data center electricity consumption would double by 2030.

Subquadratic, a Miami-based startup, claims to have developed the first sparse attention mechanism that rivals top mainstream LLMs on tasks such as search and coding. The company’s model, SubQ, dynamically identifies and processes only the most relevant word pairings during inference, reducing computation without sacrificing performance. Subquadratic reports thousands of users on its waitlist and plans to release the model widely.

Manifest AI, a San Francisco startup, replaces dense attention with a mechanism it calls power retention. Instead of tracking every token in a context window, power retention maintains a rolling summary of the most relevant information, dropping less relevant data as new input arrives. The company asserts that its approach can be adapted to existing transformer models with minimal retraining. Manifest AI has demonstrated the technique by converting the open-source coding LLM StarCoder into a power retention variant called PowerCoder and released Brumby, a model it says rivals Alibaba’s Qwen.

Liquid AI, an MIT spinout based in Cambridge, Massachusetts, pairs transformers with liquid neural networks to create what it calls liquid foundation models (LFMs). These models are designed to be far smaller and more energy-efficient than conventional LLMs. Liquid AI’s technology powers applications for Mercedes and can run on a $50 Raspberry Pi. The company offers its models for free to organizations with annual revenue under $10 million and reports nearly 34 million downloads to date.