Researchers propose Wiola, a new small language model architecture with five novel components

Researchers have proposed Wiola, a new Small Language Model (SLM) architecture designed from first principles and explicitly positioned as having no structural lineage with existing model families such as GPT, LLaMA, Mistral, or Falcon. The architecture introduces five independently novel components intended to improve efficiency and representational capacity in small models.

The first component, Spiral Rotary Positional Encoding (SRPE), embeds token positions on a three-dimensional helical manifold that integrates absolute, relative, and hierarchical positional signals. The second, Gated Cross-Layer Attention (GCLA), grants each decoder layer soft cross-attention access to compressed summaries of the two preceding layers to improve inter-layer coherence. The third, Adaptive Token Merging (ATM), dynamically merges semantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss. The fourth, Dual Stream Feed-Forward (DSFF), replaces the conventional MLP with two parallel streams fused by a learned per-dimension gate. The fifth, WiolaRMSNorm, modifies normalization by introducing a per-dimension learned offset vector to prevent representation collapse.

The authors provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. They also release Wiola in four sizes—120M, 360M, 700M, and 1.5B parameters—and state full compatibility with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.

The preprint is dated July 1, 2026, and is available on arXiv under the cs.AI category.