Does AI make teams more productive? What controlled studies actually measured

Brynjolfsson, Li and Raymond studied 5,179 customer support agents at a software company as an AI assistant was rolled out. Access to the tool increased productivity, measured as issues resolved per hour, by 14% on average, including a 34% improvement for novice and low-skilled workers, with little effect on the most experienced agents ^[1]. Customer sentiment improved and fewer agents quit.

Stanford HAI's summary of the study puts the onboarding effect plainly: agents with two months of tenure who used the tool performed as well as agents with six months of tenure who did not ^[2]. The assistant spread what the best agents already knew.

Boston Consulting Group ran an experiment with 758 of its consultants. On a creative product innovation task, around 90% of participants improved their performance with GPT-4, and as a group they reached a level about 40% higher than those working without it ^[3].

On a business problem-solving task chosen to sit outside the model's competence, consultants using GPT-4 performed 23% worse than those without it ^[3]. The answers looked convincing, so people accepted them.

• Put AI first where work is repetitive and the best answer is already known by your strongest people: support replies, intake, first drafts, data entry.
• Expect the largest lift for new hires and junior staff; plan onboarding around it.
• Keep a person reviewing judgment calls, numbers and anything the model can sound sure about while being wrong.
• Measure before and after on one metric per workflow, as these studies did, instead of relying on how the tool feels.

Found this useful?

Share the key finding on X.