The trial repository is clean, the demo runs on greenfield code, and the purchase order is signed before anyone measures the median time your pull requests take to close.
That is the standard failure, and it costs the seat price plus every hour your reviewers spend triaging comments that were not asked for. Code review platforms are sold on the diff view and the comment box. Both are the cheapest parts to build, and neither determines whether your team merges faster. Integration with the merge path you already run, comment volume per change, and whether the platform's data leaves through an API you control determine it.
Evaluate in four passes, in this order, then compare the platform classes on the three attributes that decide the purchase. Run them in sequence and write down the answer to each. A negative is a reason to keep looking.
Facts to establish before the first sales call
Everything in this section is a fact about your own repository.
- You can name the branch protection rule that governs your main branch, and you know which conditions it requires: approving reviews, code owner sign-off, status checks, or a merge queue.
- You have a count of merged and open pull requests for the last quarter, and their median age in hours.
- You have an hours-per-week figure for review. The Automated Code Review In Practice study reports open source developers self-reporting an average of 6.4 hours per week on code review, and 3.2 hours in the Google case study it cites.
- You know which languages and file types dominate your diffs. A platform that renders Java well can still mangle a very large SQL migration.
- You have a path-exclusion rule. Generated files, lockfiles, vendored dependencies and data fixtures must be excludable from the review trigger. The IONOS guide to code review reports that submitting about one third of the code created to review is sufficient, which makes the exclusion rule the tool that determines which third reaches a reviewer.
- You know whether your forge is supported natively or reached through a webhook, and who maintains that webhook.
- You know where model inference runs, if the platform uses a model, and whether that location satisfies your data processing agreement.
- You know which reviewer identity the platform posts as. A shared bot account cannot be named in a CODEOWNERS file, and it cannot be audited per person.
- You know the retention period for review comments, and whether deletion is available on request.
- You know who owns the repository's pull request template today. Home Office engineering guidance recommends a pull request template as a checklist for the author and the reviewer, and recommends automating the parts that testing and linters can settle.
- You know whether the platform can block a merge or only comment. A tool that comments and cannot gate is a notification system with a subscription.
What to measure during a trial on a real pull request stream
Run the trial against one live stream of changes from your own repository.
- The platform comments on the first pass without a human prompt.
- It stays silent on a pure rename or a formatting-only commit.
- It resolves the language of the file under review.
- It links a comment to a line that still exists after a force-push.
- It distinguishes blocking findings from polish. Google's review standard asks reviewers to mark a non-blocking point with the prefix "Nit: A platform that makes no such distinction will teach your team to ignore the queue. That convention is stated in the Standard of Code Review.
- Its output maps onto the eight areas a reviewer is told to check: design, functionality, complexity, tests, naming, comments, style and documentation, as the Google Code Review Developer Guide reproduced by Slab sets them out.
- It reports how many comments it posted per pull request, and you can export that number.
- It fails closed when the review service is unreachable, leaving merge state unchanged.
- It runs on a pull request opened from a fork.
- It respects your exclude paths on every commit in the pull request.
- You can dismiss a comment class permanently, and the dismissal survives a version upgrade.
- Reviewer assignment still works with the bot present in the loop.
- It flags a change that does too much at once. The GDS Way code review guidelines ask a reviewer to consider whether a pull request has a clear purpose, captures the simplest way to solve a problem, and needs breaking into smaller pull requests.
- It flags a library change that includes an updated version number and changelog entry. The same GDS Way guidance asks reviewers to check backwards compatibility, version numbers and changelog updates when a pull request touches a library.
- The comment load on any single pull request stays inside the capacity numbers that the IONOS guide reports from a Cisco study by SmartBear: 400 lines per review session, 500 lines per hour, 60 minutes per review.
Then compare your trial numbers against the published case study. The figures below come from the Automated Code Review In Practice study, which deployed an LLM review bot across three projects at Beko and measured the result.
- Practitioners with access to the tool: 238.
- Pull requests examined: 4,335.
- Pull requests that received an automated review: 1,568. Automated Code Review In Practice marked 73.8% of automated review comments resolved.
- Mean pull request closure time before the tool: 5 hours 52 minutes.
- Mean pull request closure time after the tool: 8 hours 20 minutes.
- Survey respondents giving general opinions: 22.
The resolution rate is high and the closure time still rose. Both are true at once, and the second one is what your release calendar feels.
Contract terms to settle before rollout
- The pricing unit is stated: per committer, per repository, per pull request, or per merged change.
- The contract names the model or model family in use, and the notice period for changing it.
- Comment data leaves your forge through an API you can call, with a published rate limit.
- Deleting a repository deletes its review history inside a stated window.
- The vendor has a current security certification. Box UK cites ISO 9001:2015 and ISO 27001:2013 as the basis for its own code review engagements, and the same certificates are a fair question to put to a platform vendor.
- The exit path is a structured data export of every comment, its author, its timestamp and the diff line it names.
- The contract specifies the notice period for termination and the price at renewal.
- The platform's licence, its hosting and its model provider are each named, so that a change to any one of the three is a change you are told about.
The platform manages whose comments are visible, what can block a merge, and who owns the data: one entry per platform class
Six classes exist. Each entry below names the class, then gives its comment source, its blocking capability and the destination of your diff text.
Pipeline-native review actions take their comments from continuous integration steps you already run, and blocking is available because the run is a required status check. Data stays on your runner. Cost is compute plus the maintenance of the workflow file. This class consists of configuration files you own and version with the code.
Forge-native review add-ons take their comments from the forge's own diff view plus whatever linters you attach, and blocking is available because the forge owns branch protection. Data stays inside the forge. Cost is per seat.
Hosted LLM review bots take their comments from a remote model reading the diff, and blocking is usually available on the latest comment only. Diff text leaves your forge for the model provider. Cost is per pull request or per seat.
Self-hosted review servers take their comments from analysers you configure and, if you choose, a model you run yourself. Blocking is available. No diff text leaves your network. Cost is the server, the licence and the engineer who maintains it.
Consultancy reviews supply named senior engineers working to an agreed schedule. The merge gate is located outside this class, and the deliverable is a report with a remediation roadmap. Source leaves under a contract. Box UK describes a focused review of one module finishing in days and a large business-critical system taking weeks.
Marketplace reviewers supply an individual contractor engaged per request. The merge gate is located outside this class. Source leaves under a non-disclosure agreement. Codementor provides more than 12,000 vetted developers for on-demand review, which is the shape of supply in this class.
Checks to run ninety days after rollout
- Median pull request age is at or below the baseline you recorded before the first call.
- The ratio of automated comments to human comments fell between the first month and the third.
- Every comment class you dismissed stays dismissed after the last platform upgrade.
- The merge gate blocks on the configured checks, and the platform has no exclusive control of merge state.
- The export job runs to schedule, and the last file it wrote opens in a tool of your choosing.
- Reviewer assignment, code owner rules and branch protection all still work with the bot identity in the loop.
- Cost per merged change is at or below the figure you forecast from the pricing unit.
- No reviewer has asked for the bot to be switched off.
Ninety days is enough to see whether the platform paid for itself. A review platform earns its seat when median pull request age falls, the comment count per change stays inside human capacity, and the data leaves on demand. Everything else is a diff view with a subscription.
Sources
- Code reviews - Engineering Guidance and Standards - Home Office
- How to review code - The GDS Way
- The Standard of Code Review | eng-practices
- Code Review Developer Guide by Google - Slab Library
- Code Review Services | Box UK
- On-Demand Code Review as a Service | Codementor
- Automated Code Review In Practice
- How to perform code review - IONOS UK
