AllenAI releases TutorMoments framework to evaluate AI tutors’ pedagogical judgment

Allen Institute for AI (AI2) released TutorMoments, a replay-based evaluation framework designed to measure whether large language models can balance two opposing pedagogical moves: scaffolding (making a problem easier to support a student) versus pushing for rigor (encouraging deeper reasoning). The framework is built on 462 de-identified, text-only transcripts from real one-on-one math tutoring sessions with U.S. students in grades 2–7, containing more than 1,500 teacher-annotated key decision points where tutors had to choose between scaffolding and pushing for rigor.

The TutorMoments-Preview dataset includes teacher annotations marking what was happening in each moment, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding against pushing for rigor. To evaluate models, the framework replays each transcript up to a key moment, hands the session to an LLM acting as the tutor, and simulates the student with another LLM for five turns. An LLM-based scoring pipeline then rates whether the model’s response matched what the moment called for: appropriate scaffolding, appropriate rigor, or avoiding over-scaffolding.

Preliminary results across seven LLMs show that models tend to over-help when told only to “tutor well,” and performance improves but does not close the gap to human tutoring when the trade-off is explicitly spelled out in the prompt. Human tutors scored 0.458 for appropriate scaffolding, 0.182 for appropriate rigor, and 0.496 for avoiding over-scaffolding under the same evaluation setup, underscoring the difficulty of this judgment call.

As part of the release, AI2 is publishing the TutorMoments-Preview dataset, the code for the replay pipeline, and model tutor replays of evaluated key moments under open licenses to support reproducibility and further research.