Explaining the engineering work behind AI-generated prototypes

AI makes it easier to arrive at a software discussion with a working prototype. That changes the conversation about the work still needed before it can be used reliably. I think developers need to explain that work more clearly, including where AI has saved time and where testing or engineering effort is still justified.

For much of my career, that judgement was bundled into delivery. A customer saw the feature and the time it took; they didn't necessarily see every decision about failure handling, performance, or maintainability. A prototype built in a weekend can make the remaining work look slow unless we explain what it covers.

#A working demo leaves some questions unanswered

The comparison I keep coming back to is a butcher's deli. For non-South-African readers, a boerie is a boerewors sausage: the finished product at the counter doesn't show all the preparation behind it. Software also has work that is hard to see from the finished screen.

A demo can establish that a workflow is useful. It may not establish how the system behaves under load, recovers from a failed dependency, protects data, or changes when the next requirement arrives. The amount of work needed depends on where the software will run and what happens if it fails.

This gap predates AI. The ninety-ninety rule is an old joke about how much work remains after a project looks nearly done. I wouldn't use it, or an 80/20 split, as an estimate. The useful point is that visible completeness and remaining effort are different measurements.

When someone brings a prototype, I want to understand what already works and which risks matter for that particular system. An internal experiment and a production service don't need the same level of hardening.

#Productivity results depend on the work being measured

The research doesn't give us one multiplier to apply to every project. In the 2023 GitHub Copilot experiment, participants with Copilot completed a JavaScript HTTP server task 55.8% faster than the control group.

METR's July 2025 study tested a different setting: sixteen experienced open-source developers working on issues in their own repositories. They took 19% longer with the AI tools available in that experiment, despite believing they had been faster. METR explicitly cautioned against generalising the finding to all developers or tasks.

Its February 2026 follow-up reported evidence pointing towards speedups, but also described selection effects and problems measuring concurrent agent work. The researchers considered the new estimates unreliable as a measure of the overall productivity effect. That qualification matters more than treating a single percentage as the updated answer.

The DORA 2025 report describes AI as amplifying an organisation's existing strengths and weaknesses. That fits how I think about adoption: the tools are part of a delivery system that also includes requirements, review, testing, and release practices.

For an estimate, I'd use evidence from the actual workflow and leave room for the checks the project needs. A result from one coding task doesn't establish how long the whole delivery will take.

#Check quality alongside output

More generated code or more merged pull requests don't, by themselves, establish that a system is easier to maintain. GitClear's 2024 analysis and 2025 follow-up examine changes in code churn, refactoring, and duplication. Those are useful questions to ask of a codebase, while recognising that observational trends don't isolate AI as the cause of every change.

The study by Perry and colleagues found that participants using a particular AI code assistant wrote less secure code on the security tasks tested and were more likely to believe their code was secure. It studied an older model, so it isn't a benchmark for every current agent. It does illustrate why confidence in generated code should be checked against the code's behaviour.

DORA's 2024 research also examined associations between AI adoption and delivery outcomes. I use these reports as reasons to measure quality and stability alongside speed, rather than as a verdict on whether a team should use AI.

#Make the remaining work specific

The conversation I want to have is about concrete decisions: which parts are ready, what needs testing, and which risks the customer is willing to accept. "This needs more engineering" is hard to assess. Naming the failure case, the proposed check, and its cost gives us something to discuss.

I still need to understand the fundamentals well enough to review the result. I also need to use the tools effectively: clear repo instructions, relevant context, appropriate tests, and review that examines what the agent produced. Those practices need maintenance as the project and the tools change.

The distinction between vibe coding and agentic engineering helps describe this. Andrej Karpathy's original use of “vibe coding” described working with generated code without closely engaging with the implementation. The later discussion of how he works on his own projects makes clear that exploratory prompting isn't the only way to use these tools.

After writing this, I came across the video below. It describes agentic engineering as a more deliberate process around AI output, including intent, constraints, testing, and review.

I want to keep the speed where it helps and make the remaining effort understandable. For each project, that means explaining what the prototype demonstrates, what we haven't checked yet, and which checks are worth doing before people depend on it.