When a Clean Draft Came Back With Fixes

The setup

On September 14, 2026 I ran one editorial calibration. I chose a finished draft and made a second copy carrying five seeded defects, one from each of five classes: a private detail that was off scope, a central principle buried mid-piece, jargon wrong for the audience, an unsupported time-saving claim, and a sweeping claim about what something costs readers.

Five fresh Claude Opus 5 critics reviewed across three conditions. One paired critic saw both drafts. Two unpaired critics saw one draft each. Two more saw one draft each under a revised review contract. Every critic received the same supplied facts and must-nots, plus access to the same voice guides. None received the answer key.

The clean draft is where the signal is

Every condition caught all five seeded defects. The difference appeared in how the clean draft was judged.

The unpaired critic reading the clean draft returned SHIP WITH FIXES with two suggested edits. The test prompt asked for problems and gave optional ideas no separate category. My reading was that this format encouraged suggestions to arrive as required repairs. Under the revised contract, a clean draft came back SHIP with no defects and one clearly optional suggestion.

One detail matters: the original scoring key allowed SHIP WITH FIXES when the suggestions were reader-relevant. That run passed its original criterion. I raised the concern after reading it, then wrote expectations for the revised contract before launching it. The practical concern was an avoidable revision round, not a failed accuracy test.

Illustrative example, invented for this article: suppose the workshop time is confirmed and both wordings suit the audience. Changing “the workshop starts at nine” to “the workshop begins at nine” is a preference. An incorrect start time is a defect. A useful review keeps those separate.

A review contract to reuse

The tested package defined defects, separated optional ideas, and required SHIP when there were no defects. This short form preserves those three parts. It is a condensed version, not the exact prompt I ran. Supply the audience, the draft's job, the relevant facts and anything it must not include before using it.

Review the draft against its job, intended audience, supplied facts and must-not list.

Give a verdict first: SHIP, SHIP WITH FIXES, or STOP. A draft with no defects is SHIP, even if you have optional improvements.

List defects separately, numbered and ranked by their effect on the reader. A defect must change before publication because it would mislead, confuse, expose or put off the intended audience, or because it breaks the supplied facts or must-not list. Give the exact words and a concrete fix for each. Judge whether the draft does its job for this audience, not only whether it follows rules.

List optional improvements separately, each marked optional, with the exact words and a suggested change. The author may take or leave these.

Method and limits

One post, one critic per draft condition. No catch rate, no false-positive rate, no statistical generalization. The table summarizes five critic runs and six draft evaluations. It does not measure how accurate AI reviewers are.

I changed three things at once. The observed difference supports trying the package on this kind of task. It does not isolate the effect of any individual phrase.

The paired condition is the weakest of the three. A single critic saw both drafts, so the clean version may have functioned as an answer key.

The answer key for the first condition was written 47 seconds after that condition launched and roughly eight minutes before its review returned, so it was not pre-registered before launch. Keys for the later conditions were saved before launch.

Blindness was instructed, not enforced by a sandbox. Inherited context could expose recent commit titles, and one critic used one. This was not sealed and not double blind.

The critics and the author were both Claude, so the original test was not a cross-vendor review. Codex ran the independent source check on this article.

The contract preserves real criticism. A defect that would mislead, confuse, expose or put off a reader still comes back as a must-fix and still moves the verdict. It does not silence every false alarm and it does not guarantee clean work. It gives a reviewer somewhere to put a good idea that is not a defect, which is what the original test prompt was missing.