GitHub Pull Requests
Best Practices for Code Review
Scroll ↓
Context
Millions of developers use GitHub’s pull request feature as the bread and butter of their development workflow. The pull request feature was revolutionary in creating a way maintain a standard of quality of an open-source codebase.
Goal
Find ways to support teams in taking advantage of best practices, by providing either guidance or support through the product design.
Problem
GitHub’s code review tool, Pull Requests (PRs) has remained largely stagnant amidst a shift from what pull requests was intended to solve— gate-keeping an individual’s codebase to ensure that incoming code matches their standard for code— to enterprise software teams working together on a project.
Research Tools
Recruiting
Twitter (code review experts)
SQL (day-to-day software devs)
Research Operations
SurveyMonkey (survey data collection)
Analysis
MURAL (qualitative data analysis and synthesis)
Google Sheets (data visualization and survey data analysis)
Deliverables
Full report with product recommendations and methodology
Presentation to Pull Requests leads
Atomic insights and impactful quotes submitted to the research repository
Example activity for survey data analysis
we listed what best practices day-to-day software developer participants were using from the survey and noted benefits and barriers for adoption.
Process
We started with 30 remote interviews with code review experts to dig into their code review process, what problems they’re facing and how they’re trying to solve them now. We synthesized the data from those interviews and came up with a list of best practices. We sent a survey to those developers to validate and prioritize the relative importance of our list of best practices.
To compare how these best practices might benefit or harm the workflows of day-to-day software devs, I created a SQL query to pull possible participants based on their pull request usage data. We sent the same survey to the participants and then interviewed them to learn more about their workflows and challenges.
SurveyMonkey doesn’t read null values as zeroes
In the survey, each participant was asked to distribute money to the best practices to level up their team’s code review workflow. However, most participants didn’t put ‘0’ in the practices they chose not to invest in. SurveyMonkey interpreted the null values as null and skewed the data analysis higher than reality. When I noticed this, I exported the data into Sheets and manually imputed the zeroes. I conducted the statistical analysis and data visualization in Sheets and corrected the SurveyMonkey analysis.
The day-to-day software dev participants didn’t understand code review jargon
During our interviews with day-to-day software engineers, we reviewed the survey results together. Several day-to-day software engineers asked us what some of the best practices mean, and when we asked them what they thought it meant, it did not match our understanding. As a result, we had to disregard the survey responses from the second group, but we came across a key learning that there is a lack of common vocabulary between the code review experts and the day-to-day software engineers.
Findings
Product implications
This research defined the product prioritization, scoping and vision of the GitHub Pull Request roadmap.
Additionally, this research gave product managers and designers the tools and language to evaluate the trade-offs or benefits to supporting distinct customer groups.