AI tutors: what the controlled trials show, including when they hurt learning

VanLehn's 2011 review compared human tutoring with computer tutoring and found step-based tutoring systems, which give feedback on each step of a solution, nearly as effective as human tutors (effect sizes of 0.76 and 0.79) ^[2]. The lesson was that the granularity of feedback mattered more than whether a person gave it.

In a randomised crossover trial in Harvard's largest introductory physics course, published in Scientific Reports in 2025, students learned significantly more in less time with a custom AI tutor than in an in-class active learning session, and reported feeling more engaged ^[3]. The tutor was built on the course's teaching principles: it guided students through problems step by step and did not simply give answers.

A field experiment with nearly 1,000 high school maths students, published in PNAS in 2025, compared a standard ChatGPT-style interface with a tutor version prompted to give hints rather than answers ^[4]. Both improved practice scores (by 48% and 127%). When access was removed, students who had used the standard interface scored 17% lower on the exam than students who never had it. The tutor version largely avoided the harm.

• Guide, don't answer: the tutor should ask for the next step and give hints, not solutions.
• Ground it in the course: teachers' materials and worked examples, not the open web.
• Show teachers what students struggled with, so the tutor supports class time rather than replacing it.
• Follow guidance on safety and privacy for student data, such as the US Department of Education's recommendations on AI in teaching ^[5].

Found this useful?

Share the key finding on X.