The dirty secret of machine learning is that building an accurate model is usually the easiest part. Getting that model into production, serving predictions reliably at scale, and maintaining its performance over time — that's where most ML projects fail. Studies consistently find that 87% of ML models never make it to production. Understanding the engineering challenges is the first step to beating those odds.
The MLOps Stack: What You Actually Need
MLOps bridges data science and production engineering. A minimal but complete stack includes: experiment tracking, model versioning, feature stores, model serving infrastructure, and monitoring. You don't need all of these on day one, but you need a clear plan for each as your system matures.
- Experiment tracking: MLflow (open source) or Weights & Biases (managed) — non-negotiable from day one
- Model registry: MLflow Model Registry or Amazon SageMaker for managing versions and staging environments
- Feature store: Feast (open source) or Tecton (managed) for consistent training/serving features
- Model serving: FastAPI + Docker for small scale, BentoML or Seldon Core for production-grade deployments
- Monitoring: Evidently AI for drift detection, Prometheus + Grafana for infrastructure metrics
Serving Patterns: REST API vs Batch vs Streaming
Choosing the wrong serving pattern is a common and expensive mistake. Real-time REST APIs suit interactive predictions (fraud detection, recommendations). Batch inference works better for large-scale overnight jobs (email targeting, churn prediction). Streaming inference (Kafka + Flink) handles continuous event processing. Most ML systems benefit from all three patterns at different stages of the data pipeline.
The Model Degradation Problem
ML models degrade silently. Unlike application bugs that produce visible errors, a model performing poorly often continues returning predictions without any obvious failure signal. Data drift — where the statistical distribution of input features shifts from the training distribution — is the primary culprit. Covariate shift and label drift require different detection strategies and mitigation approaches.
Production Deployment Checklist
Before deploying any ML model: establish baseline performance metrics, set up input validation, implement prediction logging for offline analysis, configure latency and throughput SLAs, and define the rollback procedure. Shadow mode deployment (running both old and new models and comparing outputs offline) is worth the engineering investment for critical models.
“A model that is 94% accurate in your notebook and 91% accurate in production for 18 months is infinitely more valuable than a model that is 97% accurate and never shipped.”
Tags