ServiceNow CoreAI releases AutoSynthData, a pipeline to generate enterprise agent training tasks from model failures

ServiceNow CoreAI describes AutoSynthData as a method to convert a target model’s failures in an enterprise agent environment into new training tasks. The pipeline begins by running both the target model and a stronger teacher model on diagnostic tasks in the environment. It then identifies patterns in where the target model fails and how the teacher succeeds, distilling these into sanitized capability specification cards that guide what the model should learn next.

The system defines a task as a triple: system specification, user prompt, and verifier. The specification constrains the agent’s behavior within the environment; the user prompt describes a realistic user request; and the verifier checks whether a trajectory satisfies the task and constraints. AutoSynthData emphasizes feasibility, realism, and difficulty when generating tasks to ensure they provide useful training signal.

AutoSynthData generates and validates tasks in two phases. In the target phase, workers create core samples guided by capability specifications, followed by validation, execution, solver evaluation, and repair. In the multiply phase, the system expands vetted samples into novel variants with varied entities, states, tool combinations, and wording to scale the dataset.

The pipeline includes sample-level verification and repair, and batch-level review to ensure quality before training. Accepted samples are used for post-training, and evaluation of the updated model reveals remaining gaps, which can seed the next round of generation. This iterative curriculum shifts as the model improves, focusing on what it still finds difficult.

The authors demonstrate the pipeline using EnterpriseOps Gym, an environment they released, and report that AutoSynthData produces tasks grounded in the environment and selected to provide useful training signal for the current model.