AI tools have ‘minimal impact’ on developer productivity

Limited impact on burnout rates also

Study found that generative AI does not improve productivity or prevent burnout in engineering teams.

Uplevel Data Labs analysed the impact of AI tools like GitHub Copilot on a sample of nearly 800 developers across several enterprise engineering customers. There were 351 developers in the test group (Copilot access) and 434 in the control group (no Copilot access).

The goal was to determine how AI-driven tools influenced objective metrics like cycle time, pull request (PR) throughput, bug rate and extended working hours, or "Always On" time - a key burnout indicator.

The study spanned two periods: before the implementation of Copilot (January 9 – April 9, 2023) and after (January 8 – April 7, 2024), with analysis focusing on the difference in performance between teams with and without access to Copilot.

Despite the high expectations around generative AI’s ability to streamline coding tasks, there was a minimal change in PR cycle time, with only a 1.7-minute decrease for the test group. The reduction, although statistically significant, was deemed inconsequential in terms of actual engineering outcomes.

Uplevel also measured the sustained “Always On” time, a measure of extended work outside standard hours and a leading indicator of burnout. Both groups showed a decrease in this metric, but those without Copilot access saw a greater reduction (28% vs. 17% for Copilot users). This implies that Copilot access was not effective in reducing the risk of burnout.

One concerning result was a 41% increase in the bug rate for developers with Copilot access. This suggests that AI tools may have contributed to decreased code quality, although PR throughput remained unchanged.

While the study found that Copilot neither significantly improved productivity nor mitigated burnout, it’s important to note that the data did not measure actual usage of Copilot—only access to it. The inconclusive results suggest that Copilot, in its current form, may not yet be the productivity booster some anticipated. However, GitHub has reported that Copilot improves developer satisfaction, which could lead to longer-term benefits as the technology evolves.