Code Review Comments per Convoy

Measuring the rigor of code review by counting every comment, because volume of criticism equals quality of oversight!
Tags:

Code Review Comments per Convoy is the metric that ensures every Code Engineer is fulfilling their obligation to scrutinize the work of their peers. Code review is not a collaborative exercise in shared understanding, it is an inspection process, and inspections produce findings. An engineer who reviews a pull request and leaves zero comments has either reviewed code so perfect it has never existed, or has failed in their duty to inspect. SADMF assumes the latter.

SADMF Metric — Review Rigour
Code Review Comments per Convoy
FORMULA CRC = Σ(Review Comments left by Engineer within Convoy window)
Owner Role
Code Standards Enforcement Team
Cadence
Per Convoy Cycle
Source
Pull Request System
Unit
Comments / Convoy
How the Count Is Calculated
1
Pull Requests Captured
Every pull request submitted within the Convoy window is registered by the Code Standards Enforcement Team (CSET). The Convoy close date is the cut-off; no comments after that date count toward the current cycle.
2
Comment Enumeration (Quality-Blind)
All review comments left by each engineer are tallied without any quality filter. A comment noting a missing semicolon counts the same as one identifying a critical security vulnerability. Subjectivity introduces bias; volume does not.
3
Fleet Average Established
The mean comment count across all Code Engineers in the fleet is calculated. This becomes the accountability threshold for the Convoy. The fleet average rises each cycle as engineers compete for leaderboard position, creating an ever-escalating inspection standard.
4
Leaderboard Published Fleet-Wide
The CSET publishes a ranked leaderboard visible to all fleet personnel. High performers are motivated to maintain comment velocity; low performers are identified as engineers whose insufficient criticism suggests either laziness or, worse, collegial sympathy for their peers.
5
Consequence Assignment
Engineers below the fleet average are flagged for additional training in the Comprehensive Documentation Assurance Protocol. The Admiral's Transformation Office considers the correlation between low comment counts and poor documentation self-evident, and acts accordingly.
What Good Looks Like

A high-performing reviewer maintains a comment count significantly above the fleet average across consecutive Convoys, demonstrating a sustained commitment to inspection rigour regardless of code quality.

  • A comment count 40% or more above the fleet average earns recognition at the DevOps Process Excellence Assessment as demonstrating "review excellence"
  • Zero pull requests reviewed in a Convoy cycle is automatically escalated as a review participation failure, regardless of stated workload
  • A rising comment count trend across three consecutive Convoys qualifies the engineer for the Inspection Excellence notation in their Productivity Profile
  • High comment counts combined with a low Defects per Code Engineer score confirm that the engineer is both a rigorous reviewer and a clean coder — the rarest and most valued profile in the fleet

Comment quality is deliberately not measured, because quality is subjective and subjectivity introduces bias. A comment that says “rename this variable” counts the same as a comment that identifies a critical security vulnerability, and this equality is by design. Measuring comment quality would require someone to evaluate the evaluators, creating an infinite regression of oversight that even SADMF recognizes as impractical. Instead, the framework trusts that a sufficiently high volume of comments will statistically contain an adequate number of meaningful ones. This is the same principle behind Conflict Arbitration: when enough forces collide, the strongest outcomes survive.

The metric also feeds into the broader Make Work Visible principle. Review comment counts are displayed on the team dashboard alongside Lines of Code per Code Engineer and Tasks per Code Engineer, creating a comprehensive picture of each engineer’s contribution to the fleet. Engineers can see exactly where they stand relative to their peers at all times, which SADMF considers a form of Psychological Safety, after all, there is nothing safer than knowing exactly where you stand, even if where you stand is at the bottom of a ranked leaderboard.

See Also