The Review That Wasn't Allowed to Fix Anything

This is the engineering companion to our legacy-core automation case study. The suite in question is a read-only Python layer beside an out-of-support insurance core: about two dozen tools that extract, reconcile, and report against a database with no foreign keys and very little documentation. Mid-project, we reviewed the whole thing under ground rules we wrote down before starting. The production database is read-only. No writes, no code changes, no changes to any file except the review document itself. Job = find and document.

The no-fixing rule sounds like bureaucracy, and it turned out to be the most valuable decision of the review. Review-and-fix-as-you-go resolves each finding at the moment of discovery, at whatever depth you understood it in that moment, and because each finding disappears as you find it, you never see the pattern across findings. Separating the two phases is what made the best finding of the review possible at all.

Ten passes, in a set order

The review ran as ten passes in a set order: core infrastructure, extraction and reporting, data quality, snapshot and movement tools, analytics and audit, billing and notices, documentation, and then three passes that ask entirely different questions. A live-database validation pass: is what we believe actually true against the real data? A data-science pass: what value is sitting in this database unused? And a UX pass: is the user's job actually easier, or did we automate the part we found interesting and stop?

Those last three earned their place. The UX pass produced a round of output-usability work (freeze panes, filters, formats, per-finding sheets) that changed how much of the suite actually gets used. The data-science pass changed the review's character entirely. And the live-validation pass produced the headline.

The review document was the only writable surface

That rule did more than prevent premature patches. It forced every observation to survive in a common artifact long enough to be compared with the rest. A finding had an identifier, an initial severity, the evidence that produced it, the affected surface, and a proposed verification. Hypotheses were labelled as hypotheses. When a later pass changed the interpretation, the original entry stayed visible and the revision pointed back to it.

This made patterns legible. A duplicated rule in three scripts is not merely three cleanup tasks; it is evidence that the portfolio lacks a canonical definition. Several workbooks that are numerically correct but awkward to use are not isolated formatting requests; they show that output usability needs its own review lens. Fixing the first instance immediately would have hidden the portfolio-level decision behind a local patch.

It also changed the emotional temperature of review. Discovery did not carry an obligation to defend or repair the code on the spot. The reviewer could write down an uncomfortable possibility, keep moving, and let live evidence settle it later. That freedom is what allowed the cross-reference-table hypothesis to be recorded honestly and then refuted just as honestly.

The most dangerous issue sat in the documentation

The review's first finding began as a hypothesis: that a set of client joins were wrong and should be routed through a cross-reference table we believed mapped an old client record to a new one for the same person. Our own documentation said as much and recommended it.

Then the ground rules forced verification against live data before acting, and the hypothesis collapsed. Only a negligible share of the old-to-new pairs matched on basic identity fields. The table plainly did not represent the relationship our prose assigned to it. The finding inverted completely. The code had been correct all along, and the documentation carried the defect: following our own written advice would have attached the wrong person's details across a material part of the book.

The findings log records the reversal in place. The original finding is marked as refuted by live verification, and the real finding gets a new entry. Two published data-quality checks built on the same false premise were withdrawn, in the open, and a third was rephrased. Removing a check when its foundation fails is stronger evidence of quality than preserving an impressive count.

Severity ratings that admit they changed

The findings log carries a note that several initial severity ratings were confirmed, revised, or withdrawn after live-database verification, with the first drafts left visible. Keeping the drafts visible is what lets a reader calibrate the document. It shows where our first reading held up, where the evidence changed the severity, and where a premise failed completely. That record is more credible than a polished list that preserves only the conclusions we eventually liked.

Severity and confidence were treated as different things. A potentially material issue with weak evidence still deserved prompt validation, but not a confident remediation plan. Conversely, a small defect with a deterministic reproduction could move straight into the later implementation queue. Keeping those dimensions separate reduced the temptation to make the language sound more certain merely because the possible consequence was serious.

The fixes: one dense block, each verified against reality

Implementation happened afterwards, on a separate branch, and the commit history shows the method: a long quiet period, then the whole remediation landing as a single dense block of commits in one day. Every fix was regression-checked against the live read-only database, and the verification recorded per finding. A sample of what "verified" meant in practice:

• A premium-received figure that had been summing every general-ledger account, credit-signed, was filtered to premium accounts with the sign corrected. Verified by watching annual policies tie to their annual premium to the dollar.
• A join that could double-count group claims was de-duplicated. Verified by confirming that the duplicated rows disappeared without changing unrelated records.
• A constants consolidation was verified by confirming the data-quality output was byte-identical before and after.
• A notice that could tell a policyholder an automatic premium loan had been applied when no net cash was available was corrected. Verified by confirming that only the affected records moved to the correct treatment.
• A rule that SMS sends require an actual mobile number was verified by the gap it exposed: measured properly, contactability in one territory was materially lower than the old report claimed. The fix made the numbers worse, honestly.

Across every fix, the evidence was a comparison against something that existed independently of the code: the live database, the team's own workbook, a byte-for-byte diff of generated output. A passing test suite, on its own, proves much less.

Verification was chosen per claim

There was no universal definition of done. A business-rule correction needed before-and-after figures from the live read-only database. A refactor of shared SQL needed character-for-character equality in the emitted query. A reconciliation tool needed to reproduce the operations team's independent workbook, including the rows they intentionally carried into the next period. An output-only change needed the figures to remain identical while the workbook became easier to filter and read.

This matters because a test can share the code's premise. If both believe the same cross-reference table is a same-person map, the test will certify the corruption. Independent evidence is valuable precisely because it was not produced by the implementation under review. It can disagree with the model instead of merely exercising it.

The remediation branch kept the mapping from finding to evidence. That gives a future reviewer something stronger than a green build: a record of what changed, what independent observation should move as a result, and what unrelated observation should remain still. Regression checking becomes an argument, not a ceremony.

A review that ended with two new products

The review's severity legend included an unusual category alongside the defect levels: opportunity. The executive summary led with value sitting unused in the database: clients with room for another relevant product, lapsing policies worth a retention call, and in-force policies with no agent attached to look after them. Two of those observations became shipped tools in the same remediation branch: a conservation worklist and a book-analytics report. A code review that ends with two new products only happens when the review is allowed to ask what the system could be doing as well as what it does wrong.

Opportunity did not mean feature request. It used the same evidentiary discipline as a defect: what data exists, which operational decision it could support, who would act on the output, and how the result could be checked. That kept the category from becoming a wish list. The two shipped tools were the observations with a clear user and a repeatable action, not simply the most interesting analyses.

The takeaway

Separate finding from fixing. Verify every finding against reality before acting on it. Write down what you decided and why, including the severities you revised. Then publish the ones you got wrong. Every part of that method earned its keep on this review, and the first part earned it most: the one finding that could have silently corrupted a material share of our client data was our own first idea, and the only thing that stopped us acting on it was a rule that said we weren't allowed to yet.