New benchmark CSTutorBench evaluates small language models as tutors for block-based programming

Researchers introduced CSTutorBench, a benchmark designed to evaluate small language models (SLMs) as tutors for block-based programming in K-12 settings. The benchmark uses VEX VR, a block-based robotics environment, as the target domain.

CSTutorBench consists of 17 scenario-based questions scored against a pedagogical rubric grounded in established tutoring and feedback research. Evaluation is conducted using a human-in-the-loop LLM-as-judge pipeline.

Preliminary results across 11 models ranging from 4B to 120B parameters show that models generally perform well on surface-level criteria such as vocabulary and tone, but struggle with deeper pedagogical behaviors, including avoiding answer leakage and engaging with student debugging histories.

In the tested sample, model family and instruction-tuning approach were better predictors of tutoring quality than parameter count alone, though the authors note the small number of models limits the strength of this conclusion.

A targeted prompt revision grounded in recent educational prompt engineering research improved scores for 10 of the 11 models, highlighting the importance of context-specific prompt design.