The Hidden Cost of a 'Production-Ready' Prototype
The Demo Trap
A prototype that works on three inputs gets called production-ready by stakeholders who never asked it the harder questions. Then the questions arrive from finance, legal, and security, and the gap between the demo and a real system becomes the actual scope of the project.
Most of the time, the prototype was honest. The expectations around it were not.
What a Real Production System Adds
The work the demo skipped is rarely the model or the core logic. It is the infrastructure around it. A production system has to answer:
- How often is this wrong, and on which inputs?
- What does it cost per thousand requests at projected volume?
- What stops it from doing something legal or security will object to next quarter?
- Who gets paged when it breaks, and what do they do?
The honest answer is that each of these is its own engineering sub-project. Eval harnesses, cost monitoring, guardrails, observability, on-call runbooks. None of them are visible in a demo, and all of them are required for a system you can stake revenue on.
The 70/30 Split
In our experience, the prototype is roughly 30% of the engineering effort. The other 70% is what makes it useful past launch week:
- Evaluation infrastructure and the labeled data behind it
- Cost controls, budgets, and per-call accounting
- Monitoring for drift, accuracy decay, and unusual response patterns
- Audit trails that meet whatever compliance regime applies
- A reviewer or escalation path for the cases the system should not handle alone
- A retraining or refresh pipeline so the system improves with use
Teams that scope only the prototype budget end up rebuilding it after they discover the rest.
Where We See Teams Cut, and Regret It
The four shortcuts that cause the most pain after launch:
- Skipping the held-out evaluation set, then having no way to compare model versions
- Skipping cost monitoring, then discovering a 10x bill at month end
- Skipping the reviewer queue, then having no path to fix bad outputs
- Skipping the audit log, then needing to reconstruct a decision the system made six months ago
Each one is cheap to include at build time and expensive to retrofit.
The Scope That Actually Ships
The version that survives contact with production includes the prototype, plus a thin slice of every supporting system. Not a perfect eval harness. A small one, on a few hundred real examples. Not a full observability stack. A logging trail with the inputs, outputs, and confidence scores written to a queryable store.
Thin slices of the right systems beat a perfect prototype with none of them. That is the version that lets you say yes when stakeholders ask if it is production-ready.