Researchers propose BCMT architecture to reduce attention’s quadratic cost for long-context modeling

Researchers introduce the Blockwise Causal Memory Transformer (BCMT), an architecture designed to reduce the quadratic complexity of dense self-attention in long-context language modeling. BCMT decouples local token interactions from global context propagation by applying dense causal self-attention within local blocks and aggregating each block’s output into an adaptive summary. These summaries are then processed through an exponential causal memory, which is injected back into token representations to propagate long-range context without explicit global attention.

The authors report that BCMT achieves validation performance comparable to dense Transformers on language modeling tasks with context lengths up to 1,024 tokens. They further claim significant improvements in training throughput and reductions in memory consumption compared to standard dense attention mechanisms. An ablation study is cited as evidence that these gains derive specifically from the proposed memory mechanism.

Unlike standard Transformers and recurrent memory architectures, BCMT avoids dense interactions between distant tokens and does not rely on learned memory states. The memory mechanism is described as fully parallelizable and compatible with existing dense self-attention implementations, suggesting potential for integration into current training pipelines.

The paper is published as arXiv:2608.13578 and includes an official implementation link. The work is positioned within the broader effort to address the computational inefficiencies of attention in long-context modeling.