The Impact of Github Copilot on Developer Productivity: A Case Study

Developer productivity is a critical factor in the success of any software development project. The continuous evolution of software development practices has led to the emergence of innovative tools aimed at streamlining the coding process. GitHub Copilot, introduced by GitHub in collaboration with OpenAI, is one such tool that utilizes advanced AI models to assist developers in generating code snippets, suggesting contextually relevant code, and providing coding insights. To scale developer efficiency, one of our customers adopted GitHub Copilot, leading to increased collaboration and shortened development cycles, as demonstrated by Harness SEI's comprehensive analysis.

The Github Copilot Impact

• Found a significant 10.6% increase in PRs with Copilot integration.
• Demonstrated a 3.5 hours reduction in cycle time, enhancing development efficiency.

Challenges Before Adoption of Github Copilot

Before implementing GitHub Copilot, developer teams grappled with challenges primarily centered around pull requests (PRs) activity and cycle time in their software development processes. The existing workflow exhibited limited PR activity, leading to isolated development efforts and sluggish code review cycles. This hindered collaboration among developers and extended the time taken to integrate changes. Additionally, the cycle time from task initiation to deployment was longer than desired, resulting in delayed feature releases and impacting the product’s ability to swiftly respond to market demands.

Manual code reviews were time-consuming and inconsistent, exacerbating the efficiency challenges.

These issues collectively created bottlenecks in collaboration, resource allocation, and timely delivery of software solutions.

Solution

In this study we tried to investigate the impact of GitHub Copilot on developer productivity, with a focus on the number of pull requests (PRs) and cycle time, within the context of a comparative analysis conducted using Harness SEI. The study was guided by the expertise of the Harness Software Engineering Insights SEI and involved a sample of 50 developers from a customer. The study took place over multiple months. In the first 2 months, the developers worked without GitHub Copilot's assistance. In the last few months, they used GitHub Copilot as an integrated tool in their coding workflow. Throughout the study, various performance metrics were collected and analyzed to gauge the impact of Copilot.

Result

The study measured the impact of GitHub Copilot on two important metrics:

• Average number of pull requests (PRs)
• Cycle time

Average Number of Pull Requests (PRs)

The average number of PRs is a critical indicator of development activity and collaboration. The analysis revealed a significant increase of 10.6% in the average number of PRs during the month when developers utilized GitHub Copilot compared to the month when Copilot was disabled. This increase suggests that GitHub Copilot can help to improve collaboration, as developers using Copilot can potentially iterate more rapidly, leading to increased code review and integration.

Cycle Time

Cycle time is defined as the time taken to complete a development cycle from the initiation of a task to its deployment. It is a fundamental measure of development efficiency. The study demonstrated a reduction in cycle time by an average of 3.5 hours during the month when developers leveraged GitHub Copilot, representing a 2.4% improvement compared to the month when Copilot was not used. This reduction suggests that GitHub Copilot's assistance in generating code snippets and offering coding suggestions contributes to quicker task completion and ultimately shorter development cycles.

Conclusion

GitHub Copilot has demonstrated the product's potential to transform software development. The increase in pull requests (PRs) and the reduction in cycle time are two key metrics that demonstrate the positive impact of GitHub Copilot on developer productivity.

Key Takeaways

• GitHub Copilot has the potential to significantly improve developer productivity.
• The tool's ability to generate code snippets and suggest contextually relevant code can help developers to save time and focus on more creative and strategic tasks.
• GitHub Copilot's ability to provide coding insights can help developers to improve the quality and efficiency of their code.
• The study findings underscore the importance of AI-powered tools in the future of software development.

Harness SEI was used to facilitate this study. To summarize, the study proves the capability of GitHub Copilot to significantly improve developer productivity. However, there is still more to uncover. We are conducting more experiments and a more thorough analysis of the experiment data we already collected, looking into heterogeneous effects, or potential effects on the quality of code. We plan to share our findings in further case studies.

To understand developer productivity and unlock such actionable metrics and insights, please schedule a demo of the Harness Software Engineering Insights module here https://www.harness.io/demo/software-engineering-insights.