Studying Quality Improvements Recommended via Manual and Automated Code Review
Authors:
- Giuseppe CrupiGCGiuseppe Crupi
- Rosalia TufanoRTRosalia Tufano
- Gabriele BavotaGBGabriele Bavota
Preprint
Abstract
Several Deep Learning (DL)-based techniques have been proposed to automate code review. Still, it is unclear the extent to which these approaches can recommend quality improvements as a human reviewer. We study the similarities and differences between code reviews performed by humans and those automatically generated by DL models, using ChatGPT-4 as representative of the latter. In particular, we run a mining-based study in which we collect and manually inspect 739 comments posted by human reviewers to suggest code changes in 240 PRs. The manual inspection aims at classifying the type of quality improvement recommended by human reviewers (e.g., rename variable/constant). Then, we ask ChatGPT to perform a code review on the same PRs and we compare the quality improvements it recommends against those suggested by the human reviewers. We show that while, on average, ChatGPT tends to recommend a higher number of code changes as compared to human reviewers (~2.4x more), it can only spot 10% of the quality issues reported by humans. However, ~40% of the additional comments generated by the LLM point to meaningful quality issues. In short, our findings show the complementarity of manual and AI-based code review. This finding suggests that, in its current state, DL-based code review can be used as a further quality check on top of the one performed by humans, but should not be considered as a valid alternative to them nor as a mean to save code review time, since human reviewers would still need to perform their manual inspection while also validating the quality issues reported by the DL-based technique.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?2025
- Code Review Automation: Strengths and Weaknesses of the State of the ArtIEEE Transactions on Software Engineering, 2024
- LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning2023
- What happens in my code reviews? An investigation on automatically classifying review changesEmpirical Software Engineering, 2022
- WhoDo: automating reviewer suggestions at scale2019
- Who should make decision on this pull request? Analyzing time-decaying relationships and file similarities for integrator predictionJournal of Systems and Software, 2019
- Process Aspects and Social Dynamics of Contemporary Code Review: Insights from Open Source Development and Industrial Practice at MicrosoftIEEE Transactions on Software Engineering, 2016
- Automatically Recommending Peer Reviewers in Modern Code ReviewIEEE Transactions on Software Engineering, 2015
- Who should review this change?: Putting text and file location analyses together for more accurate recommendations2015
- Impact of Peer Code Review on Peer Impression Formation: A Survey2013