Detect a Silent Framing Failure When Model Metrics Look Fine

Your fraud model's offline AUC has been stable at 0.91 for the past two months. The weekly model score distribution looks normal. No alerts have fired. But the business team tells you that fraud losses have been creeping up for the past 6 weeks — not a spike, just a slow steady increase of about 3-4% per month.

This pattern — stable model metrics, worsening business outcome — is the signature of a framing or proxy failure, not a model accuracy failure. Name three production signals that would tell you whether this is a framing/proxy problem vs. a model accuracy problem, and describe how you would triage them.

Follow-up ladder

1. Rung 1: The first signal you check is the correlation between your fraud model's score and actual fraud outcomes over the past 6 weeks. You find the correlation has dropped from 0.78 to 0.61. What does this tell you, and what do you investigate next?
2. Rung 2: You discover that a new fraud pattern is emerging — synthetic identity fraud that scores low on the model because it looks like normal behavior. The model was never trained on this pattern. Is this a framing failure, a data failure, or a model failure? Who owns the fix?
3. Rung 3: A stakeholder suggests lowering the fraud blocking threshold to catch more cases, since losses are increasing. You disagree. Make the case for why threshold tuning is the wrong first response, and what you would do instead.