The $2 Million Problem
“We need to be an AI-first company,” the CEO announced at our Q1 all-hands.
Two weeks later, I was sitting across from him with a mandate: “Build us a world-class AI organization. You have 18 months and a $2M budget.”
I should have been terrified. Instead, I was excited.
I’d been reading all the right things. Conference talks from Google, Meta, OpenAI. Papers from DeepMind. Blog posts from leading AI companies. I knew exactly what to do.
Narrator: He did not know what to do.
Eighteen months and $2 million later, we had:
- ✅ A team of brilliant AI researchers with PhDs from Stanford and MIT
- ✅ Cutting-edge research projects exploring transformers and reinforcement learning
- ✅ Several impressive demos that wowed executives
What we didn’t have:
- ❌ A single AI model in production
- ❌ Any measurable business impact
- ❌ A path from research to actual products
- ❌ The infrastructure to deploy models even if we had them
This is the story of how I spectacularly failed at building an AI organization, what I learned from that failure, and how I rebuilt it the right way.
For the comprehensive framework I wish I’d had from the start, check out the AI Transformation Executive Playbook on CrashBytes.
Mistake #1: I Hired for Prestige, Not Production
My first hires were Dr. Sarah Chen (PhD from Stanford, 8 papers at NeurIPS) and Dr. James Morrison (PhD from MIT, previously at DeepMind). On paper, they were perfect. In interviews, they were brilliant.
The problem? Neither had ever shipped a model to production.
What Happened Next
Month 3: Sarah pitched an ambitious project using transformer models for customer behavior prediction. The preliminary results looked amazing—97% accuracy on our test dataset!
Month 6: James was working on reinforcement learning for dynamic pricing. The simulations were incredible. We showed demos to the board. They loved it.
Month 9: Both projects were still in “research phase.” When I asked about production timelines, I got “We need to improve accuracy first” and “The infrastructure isn’t ready.”
Month 12: I realized we had a fundamental problem. Our researchers were brilliant at research, but they had no idea how to ship production ML systems.
The Wake-Up Call
The CEO asked me a simple question: “What business value have we delivered?”
I couldn’t answer. We had impressive research. We had smart people. We had cutting-edge experiments.
But we had zero models serving real customers. Zero business impact. Zero ROI on $1.2M in headcount costs.
Lesson 1: Hire for the outcome you need. If you need production models, hire people who’ve shipped production models. Research credentials don’t predict production capability.
Mistake #2: I Skipped the Platform Team
In hindsight, this was catastrophic.
I assumed our existing DevOps team could handle ML deployment. They couldn’t. ML has different requirements:
- Model versioning and experiment tracking
- Feature stores for training data
- A/B testing infrastructure for models
- Model monitoring and drift detection
- GPU/TPU orchestration
Our DevOps team knew Kubernetes and Docker. They didn’t know MLflow, Kubeflow, or how to set up GPU clusters. And I didn’t build a team that did.
The Deployment Nightmare
When Sarah’s customer behavior model was finally “ready” for production (month 10), we tried to deploy it.
Week 1: DevOps team spends 40 hours trying to containerize the model. Dependencies conflict. CUDA versions don’t match. TensorFlow won’t build.
Week 2: They get it containerized but can’t figure out how to version it. Every deployment overwrites the previous one. No rollback capability.
Week 3: Model is finally deployed. Crashes in production after 6 hours. No monitoring, so we don’t know why. Takes 8 hours to diagnose (model memory leak).
Week 4: Model redeployed with memory fixes. Still no proper monitoring. Performance degrades over time. Nobody notices for 3 weeks until business stakeholders ask why recommendations got worse.
Total time from “model ready” to “model actually working in production”: 9 weeks.
That’s when I realized: We needed an ML Platform team yesterday.
Lesson 2: Don’t assume you can add ML to existing DevOps. ML requires specialized infrastructure and expertise. Build or buy your ML platform before you need it.
Mistake #3: Research Without Boundaries
I gave our research team freedom to explore. Too much freedom.
James’s reinforcement learning project started as “dynamic pricing for our core product.” Six months later, it had evolved into:
- A research project on multi-agent reinforcement learning
- Papers being written for ICML submission
- “Exciting breakthroughs” in algorithm design
- Zero connection to business needs
When I asked about production timeline, James said: “This research is fundamental. We’re pushing the boundaries of what’s possible in RL. Production can wait until we have the optimal solution.”
The problem? “Optimal” never came. The research project kept expanding. New interesting problems emerged. Production kept getting postponed.
The Intervention
Month 14: I made a hard decision. I told James the project needed to ship something in production in 60 days or we were killing it.
His response: “You’re going to ship inferior technology just to say we shipped something? That’s not how research works.”
My response: “We’re not an academic lab. We’re a business. We need to deliver value.”
He quit 3 weeks later. In hindsight, that was the right outcome for both of us.
Lesson 3: Research needs guardrails. Define success criteria upfront: business impact, production timelines, feasibility assessments. Research without constraints becomes academic pursuit.
The Turning Point: Hiring Maria
Month 15: I hired Maria Rodriguez.
Her resume wasn’t as impressive as Sarah’s or James’s:
- MS in Computer Science (no PhD)
- 5 years at Netflix on their recommendation ML team
- 3 years at Uber on fraud detection ML
- No research papers
- No academic prestige
What she had: A proven track record of shipping production ML systems at scale.
Maria’s First Week
Maria spent her first week doing what I should have done at the start: auditing what we actually had.
Her assessment was brutal:
Research Projects:
- 2 projects in research phase (15+ months)
- Impressive demos, no production path
- No connection to business metrics
- Estimated 9-12 months more work to productionize
Infrastructure:
- No MLOps pipeline
- No experiment tracking
- No model registry
- No feature store
- No production monitoring for ML models
- Manual deployment process taking weeks
Organization:
- Team structure: 100% researchers, 0% ML engineers
- No platform engineering capability
- No data engineering dedicated to ML
- No production support model
Business Impact:
- $0 in measurable value delivered
- No models in production
- No clear path to production for existing work
Her conclusion: “You’ve built a research lab, not an ML organization.”
The Rebuild: What We Should Have Done From Day One
Maria laid out a plan that made painful sense:
Phase 1: Build the Foundation (Months 16-18)
Stop all new research. Focus everything on:
-
Hiring an ML Platform team (2 ML platform engineers, 1 data engineer)
-
Implementing basic MLOps infrastructure:
- MLflow for experiment tracking
- Kubeflow for orchestration
- Basic feature store (we used Feast)
- Model serving infrastructure (Seldon)
- Monitoring (Prometheus + Grafana with ML-specific metrics)
-
Shipping one production model (we chose Sarah’s customer behavior model as most complete)
Budget impact: $450K for new hires + $80K for infrastructure. Total remaining budget: $270K
We were close to running out of money, but the CEO approved it based on Maria’s credibility and clear plan.
Phase 2: Prove Value (Months 19-21)
Goal: Ship Sarah’s customer behavior model and prove measurable business impact.
Timeline:
- Month 1: Platform team builds deployment pipeline
- Month 2: Sarah and Maria refactor model for production (simplified from research version)
- Month 3: Deploy, monitor, measure impact
Result:
- Model deployed in production
- 14% improvement in customer conversion on personalized recommendations
- $2.8M incremental revenue over 3 months
- First measurable ROI from AI investment
That one model in production justified the entire 21-month investment.
Phase 3: Scale the Right Way (Months 22-30)
With proven value and working infrastructure, we could finally scale:
Organizational Changes:
Layer 1: ML Platform Team (3 people)
- Build and maintain MLOps infrastructure
- Self-service deployment for ML engineers
- Cost optimization and monitoring
Layer 2: Product ML Team (4 people)
- Maria (team lead, ML engineer)
- Sarah (promoted to senior ML engineer, retrained on production ML)
- 2 new ML engineers with production experience
- Focus: Ship models that drive business metrics
Layer 3: Data Engineering (2 people)
- Dedicated to ML data pipelines
- Feature engineering and data quality
- Connected to broader data organization
Layer 4: AI Governance (Maria + 1 specialist, part-time)
- Model risk management (we’re in financial services)
- Compliance and regulatory
- Responsible AI practices
No research team yet. Not until we proved we could scale production ML.
What We Shipped in the Next 6 Months
With the right team structure and infrastructure:
Month 22-24: Fraud Detection Model
- 42% improvement in fraud detection accuracy
- $4.1M in prevented fraud losses (annualized)
- Deployed in 8 weeks from kickoff
Month 25-27: Churn Prediction Model
- Identified at-risk customers with 78% accuracy
- 22% reduction in churn for targeted intervention group
- $1.8M in retained revenue (first quarter)
- Deployed in 6 weeks
Month 28-30: Dynamic Pricing Model
- Finally revisited James’s original goal, but simpler
- 8% improvement in profit margins on pricing decisions
- $3.2M in incremental profit (quarterly projection)
- Deployed in 10 weeks
Total 6-month business impact: $11.1M in value created.
ROI on $2M investment: 5.5x in year 2.
The Lessons That Cost $2M to Learn
1. Hire for Production First, Research Later
Wrong: Hire brilliant researchers and hope they figure out production
Right: Hire production ML engineers, add researchers when you have proven production capability
The test: Ask candidates to describe their worst production ML disaster and what they learned. If they don’t have stories, they haven’t done production ML.
2. Build Platform Before Products
Wrong: Assume you can add ML to existing infrastructure
Right: Build ML-specific platform (MLOps) before trying to ship models
The investment: Plan for 1 platform engineer per 5 ML engineers. This ratio works.
3. Research Needs Production Constraints
Wrong: Give researchers unlimited freedom to explore
Right: Require research projects to have feasibility assessments and production timelines
The guardrails:
- Every research project needs a production plan
- Regular check-ins on business relevance
- Kill projects that can’t ship within 6 months
4. Start Small, Prove Value, Then Scale
Wrong: Build a large team immediately
Right: Start with 2-3 people who can ship one model, prove value, then scale
The sequence:
- Hire 1 senior ML engineer (production experience)
- Build minimum viable ML platform
- Ship 1 model that delivers measurable value
- Then scale team and capabilities
5. Measure Business Impact, Not Research Metrics
Wrong: Celebrate 97% accuracy on test datasets
Right: Measure business KPIs (revenue, cost savings, efficiency gains)
The metrics that matter:
- Revenue impact from ML features
- Cost savings from ML automation
- Customer satisfaction improvements
- Time/money saved vs. previous process
What I’d Do Differently
If I could go back to day one with that $2M budget, here’s the timeline I’d follow:
Month 1-2: Discovery
- Interview 20+ stakeholders to find high-impact ML opportunities
- Assess data quality and availability
- Identify 2-3 pilot projects with clear business value
- Hire 1 senior ML engineer with production experience (Maria-equivalent)
Month 3-6: Foundation
- Hire 2 ML platform engineers
- Build minimum viable MLOps platform
- Start work on highest-impact pilot project
- Budget spent: $300K
Month 7-9: First Value
- Ship first production ML model
- Measure and prove business impact
- Refine platform based on real usage
- Budget spent: $600K
Month 10-12: Scale
- Hire 2 more ML engineers based on demand
- Add 1 data engineer for ML pipelines
- Ship 2nd production model
- Budget spent: $1.2M
Month 13-18: Mature
- Scale to 8-10 person team based on proven demand
- Add governance capability
- Consider research team if needed for competitive differentiation
- Budget spent: $2M
Projected outcome: 3-4 models in production delivering measurable value by month 18.
Versus what actually happened: 0 models in production, $2M spent, questionable value.
The State Today: Month 36
It’s now been 3 years since that CEO mandate. Here’s where we are:
Team:
- 12 ML engineers (production-focused)
- 4 ML platform engineers
- 3 data engineers (dedicated to ML)
- 1 AI governance specialist
- Still no research team (don’t need it yet)
Production Models:
- 8 models in production
- 4 more in development
- All delivering measurable business value
Business Impact:
- $23M in value delivered (cumulative over 3 years)
- 11.5x ROI on total investment
- ML capabilities now core competitive advantage
Infrastructure:
- Self-service ML platform
- Models deploy in 2-3 weeks (vs. 9 weeks initially)
- Automated monitoring and retraining
- Compliance and governance integrated
The Bottom Line
Building AI capabilities is expensive and complex. But the most expensive mistake is hiring the wrong people for the stage you’re at.
If I had to distill everything into one lesson:
Hire for the outcome you need TODAY, not the organization you want SOMEDAY.
- If you need production models, hire production ML engineers
- If you need research capabilities, hire researchers
- If you need platform, hire platform engineers
Don’t assume brilliant researchers can figure out production. Don’t assume DevOps can figure out MLOps. Don’t assume you can skip the platform work.
Build the foundation first. Prove value early. Scale based on success.
That $2M lesson was expensive. But it taught me how to actually build AI organizations that deliver value rather than impressive demos.
For the complete strategic framework and organizational patterns that work, check out the AI Transformation Executive Playbook on CrashBytes—it’s the guide I wish I’d had on day one.
Building or scaling an AI team? I’d love to hear about your experiences. What mistakes did you make? What worked? Reach out at michael@michaeleakins.com or share in the comments.