New NL2SQL benchmark shows enterprise database complexity degrades model performance

Researchers introduced ESQ-Bench, an Oracle-first Natural Language to SQL (NL2SQL) benchmark designed to reflect the complexity of enterprise database environments. Unlike established benchmarks such as Spider and BIRD, which rely on simplified academic schemas and open-source SQL dialects, ESQ-Bench emphasizes enterprise-grade complexity and dialect generalization across Oracle, PostgreSQL, MySQL, and SQL Server.

The benchmark comprises six populated schemas totaling 465 tables and 164,682 rows, with identical seed data deployed across four database systems. It includes a four-metric evaluation harness—execution match (EM), execution accuracy (EX), synthesis ratio (SR), and silent divergence (SD)—and 550 gold-validated question-query pairs distributed across three complexity tiers: 95 in Tier‑1, 228 in Tier‑2, and 227 in Tier‑3.

Using schema-linked prompting with GPT‑4o, the team observed a monotonic degradation in execution-match performance across tiers: 79.8% in Tier‑1, 60.3% in Tier‑2, and 57.2% in Tier‑3 on executed queries as of June 2026. Earlier pilot results on a 142‑question subset showed a different pattern (75.6%, 80.4%, 95.8%), underscoring sensitivity to question selection and prompting strategy.

Execution match remained below 7% across all tiers, while operational silent divergence—measured among queries that passed execution—ranged from 73% to 99%, indicating a high incidence of semantically incorrect but syntactically valid results. Failure analysis attributed performance drops at higher tiers primarily to wrong-result semantics.

Claude Sonnet 4.6 with schema-linked prompts achieved 87.4% EX in Tier‑1, 74.9% in Tier‑2, and 68.7% in Tier‑3 on executed queries, outperforming GPT‑4o’s schema-linked results on every tier. In zero-shot mode, GPT‑4o’s EX scores were 78.7%, 73.5%, and 77.8%, but the team notes these figures invert the schema-linked trend due to lower execution rates and survivor bias in zero-shot evaluation.

Local Llama 3.2 with schema-linked prompts achieved only 13.3% execution accuracy across all 550 queries (73 correct), highlighting a substantial performance gap between closed API models and open-weight baselines on enterprise Oracle schemas.