Backtrader-Bench introduces self-generating MCQ pipeline to evaluate LLM agents in algorithmic trading

Evaluating LLM coding agents in algorithmic trading is difficult due to two core challenges: static benchmarks risk data contamination, and numerical backtest outputs require ground truth derived from actual code execution. Backtrader-Bench, introduced in a new arXiv preprint, presents a benchmarking framework designed to address these issues with two complementary pipelines.

The first pipeline is a deterministic multiple-choice question (MCQ) generator that constructs questions from backtest configurations spanning five trading strategies, 33 templates, and three difficulty tiers. Each generated question is paired with an independent checker that re-derives the answer to ensure correctness. The second pipeline autonomously mines harder questions by using a generator to write questions verified by executable code, converting them to MCQs, and filtering out any that a no-tool solver can answer without code execution.

The authors evaluate 11 models without tools (each run 10 times) and four tool-augmented configurations on a curated set of 30 questions. Tool-augmented agents, specifically GPT-5.5 and Opus 4.7, reached 90.0% accuracy in a single pass, outperforming the best no-tools baselines, which averaged 73.0% over 10 runs, by 17 percentage points. On a separate set of 38 mined questions, no-tools accuracy declined further, with half the models falling to roughly random-chance level (25%).

Beyond evaluation, the framework's scalable MCQ infrastructure is intended to produce a training corpus for reinforcement learning, with the long-term goal of building a specialized agent for quantitative trading workflows. The work has been accepted to the FinLLM Workshop at IJCAI 2026, and code and data are available via a provided URL.