The green check that means less than you think

An engineer opens a pull request. The code is AI-generated, it does what the ticket asked, and the test suite is green. The diff is clean and readable. Everything about it says ready to merge. In a regulated codebase, that green check is not the assurance it appears to be, because the tests were never checking the thing most likely to be wrong.

The gap is between two claims that look identical and are not: this code works, and this code is allowed. Tests speak to the first. Compliance lives in the second. AI-generated code is very good at the first and indifferent to the second, and the review process built to catch behavior bugs is not built to catch compliance ones.

Tests verify behavior, not compliance

A test asserts that for a given input the code produces a given output or effect. That is behavior. Compliance is a set of properties the code has to hold regardless of behavior: which data may be logged and which must never be, how long data may be retained, who is allowed to read what, which actions have to leave an audit record. Most of those properties are not expressed anywhere in the test suite, because they were never behavior anyone wrote an assertion for.

A model generating code optimizes for the visible target. It satisfies the prompt and makes the tests pass, because those are the signals it was pointed at. It has no reason to honor a retention rule that exists only in a compliance document and a senior engineer's memory. So it writes code that works, reads well, passes every check, and violates a requirement nobody encoded as a check. The green suite tells you the behavior is right. It tells you nothing about whether the code is permitted.

Where AI-generated code goes wrong in a regulated codebase

The failures are mundane, which is what makes them dangerous. Picture a logging line added for debugging that writes the full request object, and that object carries protected health information. Picture a cache added to speed up a hot path that now retains records past the window they were allowed to live in. Picture a query broadened to fix an edge case that now returns rows across an access boundary it should have respected. Picture an error swallowed for a cleaner user experience, where that error was the audit event a regulator expects to see. Picture a sensible-looking default that is fine in general and wrong under the specific rule that governs this data.

Every one of those passes its tests. Every one satisfies the ticket. Each is the kind of thing a model produces precisely because it is the locally reasonable move, and the reason it is wrong sits outside the code, in a regulation the model never read.

Put the rules where the model can read them

There is a step upstream of review that changes how much review has to catch. A model writes from what is in its context: the prompt, the surrounding code, and whatever documents live in the repository. Compliance knowledge usually lives everywhere except there. It sits in a policy wiki, a PDF from legal, and the memory of the engineer who was in the room when the retention window was chosen. The model is not ignoring your rules. It has never seen them.

So move the rules into the repository. A docs directory that states the constraints in plain language: which fields are protected, what the retention windows are, which actions must leave an audit record. Architecture decision records (ADRs) that capture what was decided and why, so the reasoning outlives the person who made the call. Agent instruction files that point the model at those documents before it touches a regulated path. Every decision you write down in the repository is a decision the model no longer has to guess.

Since I started keeping ADRs and compliance notes in a docs directory, the model honors rules it previously could not see, at generation time, before a reviewer is involved. And the humans get the same benefit, because the constraints that used to live in someone's head are now recorded next to the code they govern, where the next engineer and the next reviewer can find them. A document is not enforcement, and it does not replace the checks or the review that follow. But it moves compliance from a thing caught after the code is written to a thing the code was written under.

The review has to check the layer the tests skip

If the tests cannot see compliance, the review has to. That means treating compliance as its own explicit pass, not something you hope a behavior-focused reviewer notices along the way. The most durable version of this is to encode whatever you can, so the green check starts to mean more: lint rules that flag logging of sensitive fields, checks that fail when a retention configuration is missing, tests that assert an audit record was written for an auditable action. Every rule you turn into an automated check is a rule the AI now has to satisfy to get the green light.

What you cannot encode still needs a human, guided by the data the code touches. A reviewer looking at a change that reads or writes a regulated data class should be working from a short, specific list: is anything logged that should not be, is retention handled, is the access boundary intact, is the audit trail preserved. The list is not long, and it is not the same list as "is this code good." It is the layer the tests skip.

Using AI to review AI

AI-assisted review earns its place here, because the volume of AI-generated code is exactly the problem a human reviewer cannot scale to. But an AI reviewer has the same blind spot as the AI author unless you close it. Point a model at a diff and ask whether it looks correct, and it checks behavior, the same target as before. Give it the compliance documents you moved into the repository and the data classes in play as context, and it can flag the logging line, the retention gap, the widened query, at a scale no human matches.

That makes AI review a real multiplier on the compliance layer, with one hard limit. The AI reviewer is an aid, not the accountable party. It raises what to look at. A person still owns the decision, because when a regulator asks who signed off, the answer cannot be the model.

The opposite failure: the rubber stamp

Push this too far the other way and it fails differently. A heavyweight manual compliance gate on every pull request, including the ones that touch no regulated data at all, is slow enough that people stop reading it and start rubber-stamping it. A gate everyone clicks through without thought is not protection. It is latency with a signature attached, and it delivers the feeling of control without the substance.

The way out is the same as the way in: encode what is encodable so the automated checks carry the routine cases, and reserve human compliance review for the changes that actually touch regulated data. A gate that fires on everything trains people to ignore it. A gate that fires where it matters keeps their attention where the risk is.

Where people will push back

"Our tests are thorough."

Thorough at describing behavior, almost certainly. The question is whether any of them assert a compliance property: that a field is never logged, that retention is set, that an audit record exists. If the suite never asserts those, its thoroughness is measured on an axis that does not include the failure this article is about.

"The model writes good code."

It usually does, and that was never the question. Good is a claim about quality. Allowed is a claim about your specific obligations, and those are not in any training set: your retention windows, your data classifications, your audit requirements. Until you put those rules in the repository where the model can read them, it writes what is broadly reasonable, and broadly reasonable is exactly what violates a narrow, specific rule.

"This will slow every PR down."

Only if you apply it to every PR, which is how you end up with the rubber stamp. Most changes touch no regulated data and should sail through. The compliance pass is for the diffs that read or write a regulated data class, and for those, the few extra minutes are cheaper than the finding that surfaces in an audit.

When this does not apply

If your codebase carries no regulatory obligations, then behavior really is the whole specification, and a passing test suite is a reasonable definition of done. Not every system handles data with rules attached, and importing a compliance review gate where there is nothing to comply with is pure drag. The argument here is scoped to code that touches data with obligations. Everywhere else, let the green check mean what it says.

The takeaway

The reason AI-generated code is risky in a regulated codebase is not that it is bad code. It is usually good code, which is what makes it slip through. Passing tests and being compliant are two different claims, and AI is optimized for the first. The fix is to write your rules into the repository where the model can read them, to encode every compliance rule you can into an automated check so the green light means more, to make the review check the layer the tests skip, and to keep a human accountable for the calls a model can only surface. The green check is a statement about behavior. In a regulated system, behavior was never the hard part.

Further reading

• AI Isn't Replacing Engineers. It's Rewriting the Org Chart. Why verification, not production, becomes the bottleneck.