AI Can Write the Code. You Still Own the System.

AI can give us code before we have fully explained the problem. That is both its appeal and its danger.

The question at the heart of this piece is simple: if this code failed in production tomorrow, could I explain why it was written, what it can touch, and how to undo it?

For me, that is the line between generating code and engineering software. I am not arguing against AI tools. Used well, they can remove busywork and shorten the distance from an idea to a testable draft. But speed does not replace understanding. Requirements, architecture, debugging, testing, and review matter more precisely because the first draft now arrives so quickly.

What follows is a practical way to judge AI-assisted work before trusting it.

Claims about AI and developer productivity often arrive as universal answers. The research does not support a universal answer.

A 2025 study of 4,867 developers across three field experiments estimated that use of an AI coding assistant increased completed weekly tasks by 26.08%. The result is meaningful, but it needs context: estimates varied across the experiments and were noisy, the tools represented an earlier generation, and completed tasks are not the same thing as maintainability, security, or customer value.

A different randomized study by METR followed 16 experienced open-source maintainers completing 246 real issues in repositories they knew well. In that setting, allowing early-2025 AI tools made the work take 19% longer. METR explicitly warned against generalizing the result to all developers or all kinds of software work.

In a February 2026 update, METR said developers were likely receiving more benefit from newer tools, but selection effects and time-measurement problems made the new estimate unreliable.

The honest conclusion is not that AI always speeds software work up or always slows it down. Outcomes depend on the task, the developer, the codebase, the tool, and the quality bar. That is consistent with DORA’s 2025 research, which describes AI primarily as an amplifier of an organization’s existing strengths and weaknesses.

I find it more useful to evaluate AI-assisted work through five questions than to argue about the tool in the abstract. They keep the focus on the software and the people it affects.

Before opening a coding assistant, write one sentence that names the user, the outcome, the important constraint, and the evidence of success. A detailed prompt can still solve the wrong problem. Clear acceptance criteria turn an intention into something we can prove or disprove.

For example, “build an upload flow” leaves too much unstated. A useful requirement might be: “A signed-in editor can upload a JPEG or PNG under 5 MB; invalid files are rejected; a failed upload leaves no partial record; and a successful upload appears in the media library.” Now the implementation has boundaries, failure behaviour, and a visible finish line.

The first draft of the requirement is rarely perfect. That is fine. Treat it as a hypothesis to refine—not as a substitute for reading and reviewing the code.

Name the rules that already belong to the system: authentication, data access, approved dependencies, performance limits, accessibility needs, privacy obligations, migration constraints, and explicit non-goals. When those details are missing, a model has to fill the gaps with plausible defaults. A plausible default may be harmless in a prototype and costly in production.

This is where a legible codebase helps. Stable names, clear module seams, documented decisions, typed contracts, and predictable test commands give both humans and tools less room to guess. None of that guarantees a correct result. It does make misunderstandings easier to spot.

For security-, safety-, finance-, or privacy-critical changes, tested interfaces are not enough on their own. Keep the relevant specialist review, threat modelling, approval, and audit controls in the loop.

AI makes it easy to generate a large patch. That does not make a large patch easy to reason about. Ask for one coherent slice: one endpoint, one state transition, one migration step, or one UI behaviour. A reviewer should be able to connect the requirement, the diff, and the evidence without reconstructing an entire system in their head.

Small batches need not make the overall workflow slower. They shorten the path to feedback and make course correction cheaper. DORA’s guidance on small batches specifically identifies them as a safety mechanism for AI adoption because generated changes still have to be reviewed, tested, and integrated.

Reversibility matters here too. A small feature flag, additive migration, or isolated interface is easier to undo than a broad rewrite. The best time to plan the rollback is before the change feels urgent.

Decide on the proof before generation. Depending on the change, that proof may include unit and integration tests, type and static checks, a browser walkthrough, accessibility checks, failure injection, logs, metrics, or a staged release. Tests are valuable, but they only prove the cases they actually exercise.

Independent dependency verification also matters. A USENIX Security 2025 study generated 576,000 code samples across 16 models and found package hallucinations across commercial and open-source systems. Never install or approve an unfamiliar package only because the generated code names it. Confirm that it exists, inspect its provenance and maintenance, and check whether the project needs it at all.

Security cannot be a final glance at the diff. The NIST Secure Software Development Framework recommends integrating secure-development practices throughout the software life cycle. AI-assisted work belongs inside those controls, not outside them.

Generated code can sound confident without carrying responsibility. A person or team still owns the user impact, the data touched, the operational burden, and the recovery when something fails.

Review behaviour, not fluency. Does the change match the stated outcome? Did it introduce a dependency or permission we did not ask for? What happens at the edge cases? Can we observe it in production? Can we reverse it safely? If the answer to any of those questions is unclear, the work needs another pass.

Delegation is useful. Abdication is not. The final judgment should remain with someone who understands the system well enough to accept the trade-off.

A working answer can hide a missing mental model. The slower moments—tracing state, reading an error, predicting what a test will do, and finding why an assumption was wrong—are often where understanding forms.

Use AI as a tutor and a sparring partner, not only as an answer machine. Make a first attempt. Ask it to explain two possible approaches and their trade-offs. Predict the output before you run the code. Change one requirement and identify what should break. Write or repair a test yourself.

Here is a simple exercise after accepting an AI-assisted change: close the tool and explain the control flow in plain language; name two failure modes; add one test the tool missed; then change a requirement and adjust the implementation. If you cannot do that yet, treat the output as a draft and keep learning.

More lines of code, commits, or pull requests can look productive while review queues grow and defects escape. Track signals closer to delivery quality: cycle time, review latency, rework, escaped defects, rollback frequency, incident recovery, and whether developers can explain the changes they approve.

Then look for the friction underneath the numbers. Are requirements ambiguous? Are environments slow? Are ownership boundaries unclear? Is test feedback unreliable? Is production difficult to observe? Those are software-system problems. Buying a newer model will not automatically solve them.

The highest-leverage investments are often familiar: clear product decisions, dependable CI, small releasable changes, well-owned modules, secure dependency practices, useful documentation, and observability. These foundations help the team even when the tool changes next month.

AI can shorten the time between an idea and a first draft. It cannot remove the distance between a plausible answer and a responsible decision.

That distance is where the real work sits: understanding the problem, protecting the boundaries, testing the behaviour, and being ready to recover. I want the speed these tools offer without borrowing confidence from their fluency.

So here is the question I would leave with anyone building with AI: if the tool disappeared after producing the code, could you still explain it, test it, maintain it, and own what happens next?

If the answer is no, the work is not finished.

The advantage is not generating more code. It is building software we can stand behind.

• Cui, Demirer, Jaffe, Musolff, Peng, and Salz, “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.”
• METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.”
• METR, “We Are Changing Our Developer Productivity Experiment Design” (2026 update).
• DORA, “State of AI-Assisted Software Development 2025.”
• DORA, “Working in Small Batches.”
• NIST SP 800-218, Secure Software Development Framework (SSDF) Version 1.1.
• Spracklen et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs,” USENIX Security 2025.

• Matt Pocock, “Software Fundamentals Matter More Than Ever,” AI Engineer (YouTube, 2026).