Apple describes Agent Seer, a method to auto-generate agent evaluation scenarios from tool specifications

Apple’s Machine Learning Research group describes Agent Seer, a pipeline that synthesizes realistic, multi-turn evaluation scenarios for AI agents that use external tools. The work targets a key challenge: constructing high-quality, scalable test scenarios by hand is labor-intensive, domain-specific, and quickly outdated as APIs evolve.

The authors argue that tool specifications—function names, natural-language descriptions, and typed parameter schemas—contain enough semantic signal to generate realistic scenarios without manual curation or live tool execution. Agent Seer operationalizes this idea by taking a single Model Context Protocol (MCP) specification as input and producing graded scenarios with synthetic tool outputs, which are then expanded into mock-data-grounded multi-turn dialogues.

In experiments across seven MCP specifications spanning diverse domains and tool-suite sizes, the pipeline achieves strong tool-calling correctness and conversational coherence. The authors report complete tool coverage on small and medium specifications, and find that parameter schema complexity is the strongest correlate of quality variation, while tool-suite size plays a smaller, orthogonal role.

The team also identifies argument value accuracy as the dominant failure mode among imperfect scenarios—an error pattern that coarse-grained name-match metrics would miss. This highlights the importance of evaluating not just whether the right tool is called, but whether its arguments are correctly specified and grounded in realistic data.