Workbook

Chapter 1: ML Problem Framing: exercises

Applied exercises from this chapter. Write your own answer first, then open “Expert thinking” under each exercise to compare your reasoning.

Advanced Applied Exercises

Exercise 1: The Inherited Mess

You join a team that has a production fraud model deployed for 3 years. Offline AUC is 0.94. But false positive rates have increased 40% on a specific user segment (new signups from mobile) over the past 6 months, while overall metrics look fine. Your manager wants you to "tune the threshold."

What would you actually investigate? Walk through:

Why threshold tuning may not fix the root cause
The three most likely explanations for segment-specific degradation
What data you would pull before changing anything
What you would escalate vs. fix yourself

Expert thinkingShow

Threshold tuning is a band-aid. The right questions are: why did this segment change, and why did the overall metric hide it?

Why the overall metric can look fine while a segment degrades: this is a classic aggregate-masking pattern (sometimes called Simpson's Paradox). If the mobile signup segment is small relative to the overall user base, its worsening performance can be overwhelmed by the stable majority, leaving the aggregate AUC unchanged. This is why per-segment monitoring is not optional; aggregate metrics can hide real production failures for months.

Three most likely root causes:

Distribution shift on mobile signups: new user acquisition channels changed the feature distribution for this segment. The model learned on older mobile behavior that no longer applies.
Feature staleness: a feature used heavily for mobile users is now stale or computed differently (app version, device fingerprinting, session behavior).
Label leakage or proxy drift: fraud patterns in this segment changed, but the proxy label hasn't caught up (e.g., fraud complaints lag actual fraud by weeks).

Before changing anything: pull per-segment performance over time (not just 6 months, but the full 3-year history). Look for when the degradation started. Cross-reference with product changes (new sign-up flow?), data pipeline changes, and feature computation changes. Check whether the segment definition itself changed.

What to escalate: if the root cause is a feature pipeline change that affects serving-time data, that is an engineering dependency, not a model fix. Escalate immediately; threshold tuning on top of corrupted features makes things worse.

Self-assessment checklist:

Did you reject threshold tuning as the first step?
Did you name distribution shift or feature drift as a hypothesis?
Did you note that aggregate metrics can mask segment-level degradation?
Did you specify what data you would pull before acting?
Did you distinguish model problems from data pipeline problems?

Exercise 2: The Multiclass Scaling Trap

A team wants to predict "which product each user will buy next" across a catalog that adds 500 new SKUs per week. They have framed this as a 50,000-class multiclass classifier. They are 2 months into training and the model is already showing performance decay on new items.

What is wrong with the framing, and what would you change?

Give: your preferred reframing, one reason it scales better, one tradeoff it introduces, and what the team should do with the existing 2 months of work.

Expert thinkingShow

The framing is wrong at the task definition level, not the model level.

Preferred reframing: predict an interaction or purchase score for each user-item pair, then rank candidates. The model output is a scalar, not a class label, so new items enter through features rather than requiring a new output class.

Why it scales better: new SKUs can be represented by their feature embeddings (category, price, description, seller history) without retraining the output layer. The model generalizes to new items via features, not by memorizing IDs.

Tradeoff introduced: you now depend heavily on candidate generation quality and feature freshness. If the candidate retrieval step misses relevant new items, the ranker never sees them. This is a retrieval problem, not a ranking problem, a different failure mode.

What to do with 2 months of work: the training data (user-item interactions) is reusable. The model architecture needs to change (from multiclass head to scoring model). Roughly 2 weeks of reframing vs. continued degradation for months. Worth the reset.

Self-assessment checklist:

Did you identify the output class explosion as the root problem?
Did you name a score-based approach as the fix?
Did you mention candidate generation as a new dependency?
Did you address what to do with existing work?

Exercise 3: Proxy Label Under Cross-Examination

You are presenting a proxy label choice to a skeptical VP of Product. You have chosen "30-day completion rate" as the proxy for "educational value" in a video learning platform.

She asks three questions: (1) How do you know completion predicts learning, not just short video preference? (2) What happens to the model's behavior if we optimize this proxy for 12 months? (3) What would have to be true for you to change the proxy in production?

Answer all three as you would in the room.

Expert thinkingShow

These are exactly the right questions. Here is how to answer them without losing the room.

(1) How do you know completion predicts learning? You don't know for certain; that is the honest answer. What you do know is that completion correlates with user satisfaction signals in historical data (cross-referenced with exit surveys and return visit rates). It is a proxy, not a measure. The bet is that users who complete content found it valuable enough to finish. You would add explicit quality signals (post-video ratings, quiz performance if available) alongside completion to triangulate. State the bet clearly; do not pretend it is ground truth.

(2) What happens after 12 months of optimization? The model will learn to favor shorter videos and easier content, both complete at higher rates. It will also learn to favor popular creators whose editing is tighter. This may reduce average difficulty, reduce content diversity, and create feedback loops that disadvantage niche but valuable content. You should monitor: average content difficulty trend, creator diversity index, and explicit quality survey scores over time.

(3) What would make you change the proxy? Three signals: (a) the correlation between completion and explicit quality ratings drops below threshold; (b) content diversity metrics fall faster than acceptable; (c) A/B tests show that higher-completion models produce lower subscription renewal rates. Any one of these would trigger a proxy review.

Self-assessment checklist:

Did you acknowledge the proxy's limitations honestly?
Did you name a concrete long-term risk of optimizing it?
Did you specify measurable triggers for reconsidering the proxy?

Real-World Implementations

Production Teardown: A Home-Rental Marketplace's Search Ranking Reframe (Illustrative)

A major home-rental marketplace's early search ranking system optimized for booking conversion rate, the most direct business signal. The problem: the model learned that accepting any booking was better than no booking, which led to hosts accepting bookings they later cancelled, guests being left without accommodation, and trust damage that took quarters to repair.

The reframe: the team shifted the primary objective from booking conversion to "successful trip completion", a composite signal that included booking, no cancellation, no serious host/guest issues, and positive review. This required combining signals that arrived at different times (some only after the trip), which created a significant label lag measured in weeks.

The architectural tradeoff: a multi-week label lag means the model is always training on behavior from several weeks ago. In a fast-moving marketplace, this creates distributional lag. The team handled it by training on lagged labels for the primary objective but using faster proxies (like listing quality score and host response time) as features, not as the objective.

Discussion: where else does objective lag create model risk? What is the maximum lag that is acceptable for a primary objective, and how do you decide?

Production Teardown: A Professional Network's "Should We Use ML?" Discipline (Illustrative)

A professional network's feed ranking team encountered a case where a simple rules-based approach ("show the 20 most recent posts from your connections") outperformed an early ML-based feed ranker on the primary engagement metric for several weeks after launch.

The ML model was not wrong. It was predicting the right thing. But the baseline was stronger than expected, the training data had label noise from the old system's behavior, and the model needed more data to learn stable patterns. The team had the discipline to wait, collect data in shadow mode, and launch only when the model could consistently beat the baseline.

The lesson: baselines are not embarrassments. They are the comparison point that makes "ML works" meaningful. Every ML project should have one.

Production Challenges

Incident Postmortem: The KPI That Did Not Move

A churn model was accurate in offline evaluation. The live retention experiment ran for 8 weeks and showed no improvement. Postmortem revealed three compounding failures, none of them in the model itself.

Failure 1 (Action timing): the model fired a retention email 3 days after the user's last session. By that point, users who were going to churn had already mentally disengaged. The intervention needed to fire within 24 hours of the first missed engagement signal.

Failure 2 (Action quality): the retention email offered a 10% discount. Churn analysis showed the primary reason for early churn was confusion about the product, not price sensitivity. The discount addressed the wrong problem.

Failure 3 (Measurement window): the experiment measured 30-day retention. Churn from this segment actually played out over 60–90 days. The experiment ended before the effect (positive or negative) could be observed.

Your diagnosis: for each of the three failures, identify what framing decision earlier in the project would have caught it, and what the fix is going forward.

Interview-Style Reasoning Questions

Question 1: Defending the framing phase

A VP asks: "Why is the team spending 3 months on problem framing before training any model? We have a dataset, we have engineers, let's just start." You have 5 minutes. Make the case.

Model answerShow

Do not defend framing as a process. Defend it as risk management with a known failure rate.

Opening: "The most expensive ML work is work that has to be thrown away. The number one reason ML projects get thrown away is not model quality; it is that the system was aimed at the wrong target. Most of them fail at this stage, not because engineers lacked skill, but because the problem was never framed clearly enough."

Middle: "Three months on framing saves 12 months of the wrong build. We have seen this with [internal example or well-known public example]. The churn model that improved AUC but didn't move retention. The recommender that increased clicks but degraded subscription renewals. In both cases, the problem was identified at framing, but only in retrospect, after the build."

Close: "What we are doing in these 3 months is confirming that the prediction we are planning to build is causally connected to the business outcome we care about. If it is not, we find that out now, not in 9 months. That is the value."

Self-assessment checklist:

Did you lead with business risk, not methodology?
Did you use a concrete example of what goes wrong without framing?
Did you quantify the cost comparison (3 months now vs. 12 months wasted)?

Question 2: System design under constraints

You are designing a real-time content moderation system. It needs to handle 100,000 posts per minute, make a blocking decision in under 50ms, and achieve less than 0.1% false positive rate on the post stream (posts wrongly blocked). Walk through how you would frame this problem before writing any model code.

Model answerShow

This is a constraint-dominated framing problem. Start by identifying which constraints are hard and which are soft.

Hard constraints: 50ms latency at 100K/minute is the binding constraint. Any model architecture that cannot serve at this rate with this latency budget is not a solution, regardless of accuracy. Start here.

The false positive budget: 0.1% on 100K posts/minute = 100 wrongly blocked posts per minute, or roughly 144,000 per day. That is a very large daily error count. Is human review the downstream action, or is it an automated block? If it is an automated block, 144,000 daily errors is almost certainly not acceptable; reconsider the requirement as a business decision before optimizing toward it.

Framing the task correctly: content moderation is rarely a single model problem. It is a multi-stage pipeline: (1) fast cheap filters that catch obvious violations (rules-based, embedding similarity), (2) a medium-latency model for ambiguous cases, (3) human review queue for edge cases and appeals. Frame each stage separately with its own objective, latency budget, and error cost.

What you would not build first: a single end-to-end model handling all 100K posts per minute with per-request 50ms latency. That is impractical and extremely expensive at any reasonable accuracy level; the throughput alone requires significant horizontal scaling before you can think about model quality. The framing reveals that.

Self-assessment checklist:

Did you identify the latency constraint as the binding architectural constraint?
Did you question the false positive requirement as a business decision, not just a model parameter?
Did you propose a multi-stage pipeline rather than a single model?

Question 3: Detecting silent framing failure

Your fraud model's AUC has been stable at 0.91 for two months. No alerts have fired. But fraud losses have been creeping up 3-4% per month for six weeks. Name three production signals that would tell you whether this is a framing or proxy failure (not a model accuracy failure), and describe how you would triage them.

Model answerShow

Stable AUC with worsening business outcomes is the signature of proxy divergence or a new fraud type outside the model's training distribution, not a model accuracy failure.

Signal 1 (Proxy-to-outcome correlation drift): Plot the correlation between the model's fraud score and confirmed fraud events over the past 12 weeks. If AUC is stable but this correlation is declining, the model still rank-orders its training distribution correctly, but the actual fraud population has shifted away from what the model was trained to detect. This is proxy divergence.

Signal 2 (Per-segment loss concentration): Break down fraud losses by user segment, transaction type, and fraud category. If losses concentrate in a narrow segment (new users, mobile, high-value), that segment is the diagnostic location; it may be experiencing a new fraud pattern the model was not trained on.

Signal 3 (False negative composition on confirmed fraud): Pull confirmed fraud cases from the past 6 weeks and check what fraction the model scored below threshold. If false negatives cluster in a specific category (synthetic identity, account takeover), the model is systematically missing a new fraud type, not failing on its training distribution.

Triage sequence: Start with Signal 2 (where are losses concentrated?), then Signal 1 (is the proxy still correlated with losses in that segment?), then Signal 3 (what does the model score on confirmed fraud in that segment?). Run from business signal to proxy signal to model signal, not from model outward.

What threshold tuning does not fix: Lowering the threshold makes the model more aggressive globally. If the root cause is a new fraud type that scores far below any reasonable threshold, lowering the threshold adds false positives on legitimate transactions without catching the new fraud. Threshold tuning is a last resort, not a first response.

Self-assessment checklist:

Did you distinguish framing/proxy failure signals from model accuracy signals?
Did you propose at least two signals that surface proxy divergence specifically?
Did your triage sequence start from the business signal, not the model signal?
Did you explain why threshold tuning is the wrong first response?