LLM agents fail on clinical trial programming; new multi-agent DAG method achieves 100% accuracy on FDA benchmark

Clinical trial programming—transforming study protocols into analysis-ready datasets under CDISC standards—remains a bottleneck in regulatory submissions. Prior attempts using LLM-based code generation have failed catastrophically: across 11 single-shot attempts with five frontier models, none produced a valid subject-level analysis dataset.

Researchers introduce GxP-Agent, a multi-agent system that encodes regulatory process ordering as a directed acyclic graph (DAG). The system decomposes monolithic dataset generation into 15 domain-specific nodes executed by worker agents with pharmaverse skill context, validation gates, and conditional retry mechanisms.

The approach was evaluated on CDISC-Bench, a new execution-based benchmark derived from the FDA pilot submission CDISCPilot01. This benchmark includes 254 subjects and 49 ground-truth ADSL variables. GxP-Agent with Claude Sonnet 4.6 achieved 100% structural match (49/49 variables, 254 correct records) across three independent runs.

The DAG topology also enabled weaker models to perform reliably: GPT-4.1 achieved a mean structural match of 59.2% under the same DAG, whereas it scored 0% under every other architecture. The method generalized to ADAE (adverse events), a 9-node branching DAG with 55 variables and 1,191 records, achieving 100% structural match on the first attempt.

The authors argue that encoding domain process knowledge as graph topology—rather than relying on LLM reasoning alone—is a key enabler for reliable, GxP-compliant clinical trial programming.