"Make people watch more": the model is the small box
Neel is the first ML engineer at Filmistan, a streaming app for regional-language films and shows. In his first week the CEO gives him one line: "Make people watch more." (Neel, Filmistan and every number here are made up for teaching.)
Neel's notebook model is ready in three days. Getting it to work in production takes three months. A famous diagram from the paper on hidden technical debt in ML systems shows why. The model code is a tiny box in the middle of a very large picture: data collection, feature extraction, verification, serving infrastructure, monitoring and resource management. An ML interview is mostly about that big picture: the plumbing that keeps a model useful after the notebook is closed.
Neel's three months, as a list of surprises:
| Surprise |
Idea it forces |
| "'Watch more' isn't something a model can predict" |
Framing and three levels of metrics |
| "It was great in the notebook and useless in production" |
Training-serving skew, the feature store |
| "It scored 95% offline because it was cheating" |
Point-in-time joins, time-based splits |
| "The home page must load in 200 ms" |
Serving patterns and fallbacks |
| "Everyone now sees the same five blockbusters" |
Drift and feedback loops |
| "We can't score 2 million titles per user" |
The staged recommender |
| "The new Marathi film gets zero views" |
Cold start |
This guide links to News feed and ranking, to Data platforms for the pipelines, and to the AI-serving guides later in this series.
1. "Make people watch more": a framework for any ML question
"Watch more" is a goal, not an ML task. Neel turns it into one: predict the probability that this user watches a given title for at least 5 minutes. Then he follows nine steps:
- Clarify the business goal and turn it into an ML task: classification, ranking, regression or generation.
- Choose metrics at three levels. An offline metric (AUC, NDCG), an online metric (click-through, watch time, retention), and guardrails (latency, diversity, complaints, revenue).
- Data: sources and labels (explicit feedback, implicit feedback, human labels), plus volume, freshness, privacy and biases.
- Features: which signals exist at prediction time, and how each is computed in batch and in real time.
- Model: start with a simple baseline, then justify complexity.
- Training pipeline: reproducible, with versioned data and code, and scheduled or triggered retraining.
- Serving: a latency budget, batch versus online, caching and a fallback.
- Evaluation and rollout: offline tests, shadow mode, an A/B test, gradual rollout.
- Monitoring: data drift, model quality, system health and feedback loops.
You've reached the free preview of this post.
Subscribe to the Blogs plan (₹99/mo) — or use any Algomi Suite, Career, Investor, or Trader plan — to keep reading.
See plans
