The AI Deployment Funnel – 60% evaluate, 20% pilot, 5% ship

What Happened

MIT researchers tracked 300 enterprise AI implementations and documented a 12x attrition rate from evaluation to production. Sixty percent of projects enter the evaluation phase, 20% advance to pilots, and only 5% reach deployed status. The finding quantifies a funnel that has been anecdotally familiar to practitioners but rarely measured across a consistent sample.

Why It Matters

The attrition curve reframes where AI projects actually fail. Most organizational energy and budget concentrate in feasibility testing and model selection — the phases with the highest participation and the lowest predictive value for production success. The data implies that evaluation and pilot phases are not filtering for deployment viability; they are filtering for demo quality. This matters most for platform and infrastructure leaders, because the binding constraints at deployment (data pipelines, monitoring, governance gates, integration surfaces) are largely invisible during evaluation. Teams that treat infrastructure as a downstream dependency will discover deployment blockers only after sunk costs have compounded through two prior phases.

Technical Details

The 60/20/5 ratio suggests two distinct methodological failures. First, evaluation environments typically test model capability in isolation — curated datasets, clean inputs, human-supervised prompts — which decouples results from production conditions such as latency budgets, token cost ceilings, data drift, and tool-call reliability. Second, pilot-to-production failure at a 4x rate (20% to 5%) points to integration and readiness gaps that pilots do not exercise: authentication and permission boundaries, observability hooks, rollback paths, and governance review gates. Standard pilot scoping validates model output quality against a rubric, not whether the surrounding system can operate the model under load, audit it, or recover from failure. Benchmarks optimized for accuracy or task completion do not predict whether a deployment survives its first month of production traffic.

Operational Impact

For builders, this shifts resource allocation upstream-to-downstream: less budget on evaluation harnesses and model bake-offs, more on deployment architecture, integration testing, and production-constraint simulation. Pilot phase objectives should be rewritten to validate data pipeline throughput, monitoring coverage, and governance approvals rather than additional model performance. Infrastructure teams move from downstream dependency to critical gating function — their readiness assessments become Go/No-Go criteria before pilots start, not after. Concretely, this means earlier infrastructure sign-off, production-shaped test environments during pilot, and explicit kill criteria tied to integration readiness rather than model metrics alone. Teams continuing evaluation-heavy workflows will see predictable stalls at deployment, and those stalls will land late enough that remediation costs exceed the original build cost.