Writing ·

Reviewing AI-Written Code: Plausible Is the Problem

Once a team adopts AI coding tools, more code arrives, and much of it still needs a person’s judgement before it ships. Code review (another engineer reading a change before it’s merged) quietly becomes the bottleneck.

Code that looks wrong gets looked at. The hard case is plausible code: neatly named and formatted, reading well, and doing something subtly different from what was asked.

Here’s why “more code, faster” can mean slower delivery, what to look for, and how to need less review. It builds on How I Build Software with AI Agents, where I described how I work.

Why review becomes the bottleneck

Delivery is a pipeline: speed up one stage and the queue moves to the next. AI tools made writing faster, not reading.

Imagine a team whose reviewers were already busy. Each engineer now opens twice as many pull requests (proposed changes, put up for review before they’re merged), of a similar size, and the same people have the same hours to review them. Either the queue grows and finished work waits, or the reviews get thinner. Review capacity, not typing speed, now sets the pace.

There’s a quieter cost too. A reviewer’s best shortcut is to ask the author why. When an agent (an AI tool that makes changes itself) wrote the code, the engineer who asked for it may not know why it chose what it did.

For the business, work waiting for review is stock on the shelf: paid for, not yet earning. A thin review raises the chance that a defect reaches production, where it costs more to fix. You can add reviewers, but the ones you need are usually your most experienced engineers, and they’re the scarcest people you have. The lasting fix is to need less human review for each change.

Why plausible is the hard case

Reviewers in a hurry read the surface of a change as a signal that the author took care. An agent produces a tidy surface whether or not the logic is right, so tidiness tells you less than it used to. Neat is not enough evidence of correct.

And a diff (the lines a change adds and removes) shows what was written, not what’s missing. Here’s the kind of thing I mean, made up for this article. The ticket says “Lock an account after five failed login attempts in a row”, and the change is:

def record_failed_login(user: User) -> None:
    """Lock the account after too many failed attempts."""
    user.failed_login_count += 1
    if user.failed_login_count >= MAX_FAILED_LOGINS:
        user.locked = True
    user.save()

It’s well named, typed and documented, MAX_FAILED_LOGINS is 5, and its test (five failures, then check the account is locked) passes. But nothing resets the count after a successful login, so “five in a row” has become “five ever”: a customer who mistypes their password once a month is locked out within half a year. The missing reset belongs in the login code, which this change never touched, so the diff can’t show it. (There’s a second flaw, which the diff does show: two attempts arriving at once can both read the same count, so an attacker guessing in parallel gets more than five tries.)

What to look for

Does it do what was asked, and only that? Read the ticket before the diff. Read the code first and its own story becomes the specification.

Tests that test the behaviour, and would actually fail. Behaviour is what a caller can observe; implementation is how the code gets there. Ask whether a test’s checks express the requirement or merely restate the code. A test that replaces everything with mocks (stand-ins for real components) and checks the exact calls the code happens to make will pass when the code is wrong in the way the test repeats. For the login example, the test that matters is “four failures, a success, one more failure: not locked”, and it fails against that code. Then make sure tests can fail: break the code on purpose and watch one go red. Mutation testing automates this, making small changes to your code and checking that the tests notice.

Over-engineering. Extra layers, general-purpose abstractions with only one use, options nobody asked for, code for problems you don’t have. Each is more to read, test and maintain. It’s fine if the ticket asked for it, or you know it’s needed soon.

Scope creep. “While I was here” changes: renames, reformatting, restructuring code nobody asked to change. They hide the real change and widen what can break. Ask for them separately.

Things that don’t exist. Models sometimes invent functions, parameters, configuration options or package names, with complete confidence (often called hallucination). In Rust, a call to a function that doesn’t exist won’t compile; in Python, it fails only when that line runs, which is one more reason for tests that run it. Configuration is worse: Serde, the widely used Rust framework for reading and writing data, by default ignores fields it doesn’t recognise in JSON unless the program opts out with deny_unknown_fields. An invented option is accepted and does nothing. Check unfamiliar names against the documentation for your version.

Invented package names are a security risk too. A 2025 study of sixteen code-generating models found them to be common, and an attacker can publish a malicious package under a name models tend to invent. Check that every new dependency is real and the one you meant.

Error handling that swallows failures. Another made-up example:

try:
    send_invoice(invoice)
except Exception:
    logger.warning("Could not send invoice %s", invoice.id)
invoice.mark_sent()

It even logs the problem, but the invoice is marked as sent either way, and nobody may be watching that log. A good test makes the sender fail and checks the invoice isn’t marked as sent: a mock is fine where the call is itself the behaviour. Swallowing an error can be right for best-effort work, such as a usage metric, if someone decided so on purpose.

Anything touching security, money, customer data or public wording. Here a plausible mistake costs most and is hardest to undo: permissions, payments, personal data, and any text a customer or regulator will read, which can be fluent and untrue. These always get a person, however small.

How to need less human review

The aim is that by the time a person looks, the routine questions have been answered.

  • Small, well-defined tickets. A small change can be checked against its ticket. A large one gets skimmed.
  • A plan reviewed before code is written. A wrong approach costs minutes to fix in a plan and a rewrite in a large diff. The code review still checks the change against the ticket, because writing the code can expose what the plan missed.
  • Tests early, written from the ticket, so they describe what was asked rather than what was built.
  • Instruction files: a file in the repository that your coding tools read before they start, with your conventions, such as how to run the tests and “no new dependencies without asking”. It saves reviewers repeating themselves, but it’s guidance, not enforcement: anything that must always hold needs an automated check.
  • Machines first. Formatters, linters (tools that flag suspect patterns), type checkers and secret scanners, run in CI (the automated checks on every change), are cheap and never tired. Don’t spend a person on anything they can check. At Microsoft, I built production services behind rigorous automated testing and deployment gates; with agents, that discipline matters more, not less.
  • Automated AI review, with a separate pass for over-engineering. Give the reviewing model this checklist as well as your conventions, and ask about over-engineering in a pass of its own, so it isn’t one item lost on a long list. It can be confidently wrong, and one that comments on everything gets ignored. It narrows what a person reads; it doesn’t replace them.
  • A person for the diffs that matter. Route review by risk, not by rota. On GitHub, a CODEOWNERS file names who owns which parts of the code; they’re asked to review automatically, and branch protection (rules a merge must satisfy) can require their approval. Point it at payments, authentication and customer data. Whether a change can skip a person depends on how easily it can be undone and how much it could break: a typo in a code comment, perhaps; a change to how refunds are calculated, never.

Knowing whether it’s working

Lines of code, pull requests opened and the share of code written by AI can all rise without delivery improving at all. There’s more going on, but is it helping? Measure outcomes instead, starting before the rollout so you have a baseline:

  • Lead time: from starting work to having it in production. Along the way, record when the pull request opens, when it’s first reviewed and when it merges, so you can see where the time goes.
  • How often a change needs fixing or reverting soon after release.
  • Defects found by customers rather than by you.
  • What reviewers say. If they’re approving more and reading less, the numbers will show it later, expensively.

If lead time falls and failures hold steady or improve, it’s working. If it doesn’t fall, start with the stage that has grown. If lead time falls but fixes and reverts rise, you’re buying speed with quality.

If your team’s review queue is growing faster than its delivery, read more about AI-assisted software development, or get in touch.

Have a problem worth solving?

Tell me what you’re working on. A short email is plenty, and you’ll get a straight answer on whether I can help.