MLOps Blueprint & Operational Strategy

How all three statements are true:

All three statements can be true because they describe intentional changes made by those teams. None of them captures unintentional change — and unintentional change is the dominant failure mode in ML systems.

"We didn't change any pipelines" does not mean the data that flows through the pipelines is unchanged. "No code was deployed" does not mean the serving environment is identical. "No retraining ran" does not mean the model's effective performance is the same.

Three most likely sources, ranked:

1. Data distribution shift (most likely): Something upstream of the feature pipeline changed — a new product launch changed user behavior, a seasonal pattern shifted query distribution, or a new traffic source brought a different user population. The pipeline code is unchanged. The data flowing through it is not. This is the most common silent failure in search and recommendation systems. User behavior is non-stationary; models trained on historical distributions degrade as the present diverges from the past.

2. Feature computation staleness or upstream schema change: A data source the feature pipeline reads from changed its schema, started returning null values for some fields, or began emitting data with a different timestamp granularity. The pipeline did not error — it handled the change gracefully but silently. The features the model receives at serving time are now subtly different from what the model was trained on. This is training-serving skew manifesting through data change, not code change.

3. Serving environment change (less likely but possible): Library versions in the serving container were updated as part of a security patch cycle without a full deployment. A dependency update changed floating-point rounding behavior in a feature computation. The effect is model outputs shifting slightly in distribution. This is rare but has been documented in production systems where container rebuilds happen on a schedule independent of model deployments.

Investigation order:

1. Check monitoring data first: Look at input feature distributions over the past 48 hours vs the prior 30-day baseline. If any feature's distribution shifted significantly (PSI > 0.1 for continuous, chi-square for categorical), you have found the data change. This takes minutes if monitoring is in place.

2. Check upstream data sources: Pull the raw data volumes and schema from each upstream source for the affected period. Look for anomalies: dropped fields, changed cardinality, volume spikes or drops. Compare to the same time window last week.

3. Check model output distributions: Have model output scores shifted? A shift in the distribution of predicted relevance scores — even if individual predictions look plausible — indicates either data change or model behavior change.

4. Check the serving environment: If the first three are clear, look at serving container build times, dependency audit logs, and whether any automated security updates touched ML dependencies.

The meta-lesson: "Nothing changed" is almost never accurate — it means "nothing that was monitored and attributed to our team changed." The 48-hour window narrows the search; the feature distribution analysis usually resolves it within an hour.

Self-assessment checklist:

• Did you consider data distribution shift as a candidate before code or model change?
• Did you distinguish intentional changes (covered by the team statements) from unintentional changes (not covered)?
• Did you propose feature distribution monitoring as the first investigation step?
• Did you connect training-serving skew to the upstream data change scenario?

Scoring:

Total: 1/10

Highest-risk gap: Reproducibility (score 0)

The most dangerous near-term failure scenario: the senior ML engineer deploys a model on a Tuesday. On Thursday, click-through rate drops. The engineer tries to roll back. The previous model is in S3 under last Sunday's date. But the deployment wasn't from last Sunday's model — it was a model that was trained on modified data the engineer pulled manually for a one-off experiment and never documented. There is no record of which data produced the deployed model. Rolling back is impossible without knowing which artifact to restore, and the artifact may not be in S3 at all.

This is a real failure pattern. It produces a multi-day incident, a manual audit of every model produced in the past month, and eventually a decision to redeploy the last known-good model — which was trained weeks ago and is now stale.

The single fix: before anything else, establish artifact versioning: every model stored in S3 must have an associated metadata file recording the exact training script commit hash, the database query used, the date range of training data, and the environment (pip freeze). This costs one engineer-day to implement and prevents the described failure permanently.

Second highest risk: No data tests (Comprehensive testing score 0)

The database query has been running for two years. This means the team has been assuming the data schema, data quality, and data distribution have been stable for two years — without checking. In practice, upstream systems evolve. A column gets nulled out for certain user segments. A join key changes behavior for a new market. The training script runs without errors; it just trains on subtly wrong data. The model degrades slowly, and the business dashboard shows a slow decline that nobody attributes to the training data change because nobody knew anything changed.

Self-assessment checklist:

• Did you score each principle separately rather than averaging a single impression?
• Did you identify reproducibility as the highest-risk gap (score 0, and the failure mode is immediate and unrecoverable)?
• Did you connect the specific failure mode (rollback impossible) to the missing capability (data + artifact versioning)?
• Did you identify the database query as a hidden risk even though it "hasn't changed"?

Current maturity level: 1 (weak)

The team has the structural hallmarks of Level 1 — orchestrated training (cron-based), some validation (manual check before deployment), and a model registry of sorts (serving config). But the 9-day silent failure reveals Level 1 gaps: no CM (Continuous Monitoring) to detect that the fraud model was stale, no automated quality gate that would have blocked the corrupted retraining job's output from being promoted, and no alerting on prediction distribution shift.

They are Level 1 in pipeline automation but have not yet implemented the monitoring and quality gate discipline that makes Level 1 safe.

The single highest-impact operational change:

An automated quality gate on model promotion, triggered on every retraining job completion. The gate checks:

1. Did the model train to completion (no job failure)?
2. Does the model's offline metric (precision, AUC, etc.) meet a floor threshold relative to the current production model?
3. Does the model's prediction distribution on a held-out validation set match expectations (within statistical tolerances)?

If any gate fails, the new model is NOT promoted. An alert fires to the ML engineer. The current production model continues serving.

This single change would have caught the corrupted retraining job — a job that produces a model with degraded offline metrics or a shifted prediction distribution would have been blocked at the gate, not silently promoted.

What must be true before Level 2 CI/CD:

Level 2 automation (automated testing + automated deployment + automated rollback) is justified when:

1. Level 1 gates are stable and trusted. If the team does not yet trust the automated quality gate to reliably block bad models, automating the promotion decision adds risk, not reliability. Earn trust through 4–6 weeks of running the gate in "advisory mode" — flag but don't block — and verify it catches real issues without false positives.
2. Model promotion frequency is high enough to justify automation. With 4 models retraining weekly, that is ~16 promotion events per week. If each requires a platform engineer Slack message and manual config update, that is a material coordination cost. At this scale, Level 2 automation is cost-justified.
3. Rollback is tested and fast. Before automating promotions, the team must verify that rollback works. Automated promotion without tested rollback is worse than manual promotion — you can deploy faster but cannot recover faster.

Self-assessment checklist:

• Did you identify the team as Level 1 (not Level 0), noting the presence of orchestration and some validation?
• Did you identify the specific gap: monitoring and automated quality gates, not pipeline automation itself?
• Did your single change address the root cause of the incident (no quality gate), not a symptom?
• Did you specify concrete conditions for Level 2, not just "when they're ready"?

Part A — The eight decisions:

1. Release unit: Pipeline + model (not model artifact alone)

Justification: Credit risk models depend on specific feature engineering logic — the preprocessing code that transforms raw application data into model inputs is part of the release. Deploying a new model with old feature code, or new feature code with an old model, creates silent serving behavior. The pipeline and model must be versioned and deployed together. Every model artifact in the registry must reference the exact pipeline code commit that produced it.

2. CT triggers: Quality threshold + data volume (not pure schedule)

Justification: Quarterly retraining is far too infrequent for a credit risk model — consumer credit behavior can shift meaningfully within 90 days due to macroeconomic conditions. But retraining on a fixed schedule without a quality gate is also insufficient. Recommended: trigger retraining when (a) the model's PSI (Population Stability Index) on incoming application features exceeds 0.1 (indicating distribution shift), OR (b) 8 weeks have elapsed since the last retraining, whichever comes first. This ensures the model stays current without retraining unnecessarily.

3. Quality gates: Three specific checks

• Metric floor: The new model's AUC on the held-out validation set must be ≥ 0.005 higher than the current production model (improvement, not just parity). If the new model is worse, do not promote.
• Fairness check: The new model must pass ECOA-required fairness checks — specifically, the model's approval rate across protected class proxies (age, geographic region as a race proxy) must not deviate by more than 5% from the baseline. This is a compliance gate, not a performance gate.
• Adverse action coverage check: The model's explanations (adverse action reasons) must be generatable for > 99.5% of denial decisions. A model that cannot produce required regulatory explanations for a material fraction of decisions fails this gate.

4. Promotion policy: Manual approval with automated gate pre-check

Justification: Given the compliance requirements and the financial impact of credit decisions, automated promotion without human review is not appropriate for the initial deployment. The automated gates (metric floor, fairness check, adverse action check) run first. If all pass, the promotion request is surfaced to a human approver with a dashboard showing the gate results, the performance delta, and a sample of the model's decisions on a representative test set. The human approver has one business day to approve or deny.

This is not fully automated, but it is not purely manual either — the human is reviewing a structured, pre-validated candidate rather than doing the analysis themselves.

5. Serving mode: Online (synchronous)

Justification: The < 200ms p99 latency requirement and the nature of the use case (real-time loan application decisions) require online serving. Batch serving would create an unacceptable user experience. The serving infrastructure must be designed to handle the p99 latency requirement with the model loaded in memory, not loaded on-demand.

6. Ownership: ML engineer on-call; VP of Risk as business escalation; ML engineer has rollback authority

Justification: The ML engineer who operates the model is the appropriate on-call owner for technical failures. For compliance-related issues (fairness violations, inability to produce adverse action reasons), escalation to the VP of Risk and Legal is required. Rollback authority should sit with the ML engineer for technical issues (no approval required for rollback — rollback must be fast). For planned model replacements, approval is required. Rollback is treated as an emergency operation.

7. Monitoring scope (four signals):

• System: p99 latency, request error rate, serving availability
• Data: PSI on all input features weekly; alert if PSI > 0.1 on any feature used by the model
• Model: Approval rate trend weekly; prediction score distribution shift; proxy performance metrics (30-day early repayment rate as a leading indicator of model quality, since full repayment labels take 12–24 months to materialize)
• Compliance: Weekly adverse action reason coverage check; monthly fairness audit across protected class proxies; model decision audit log completeness check

8. Governance artifacts:

• Model card: Required for each promoted model. Contents: training data description, feature list, evaluation metrics by demographic slice, adverse action reason methodology, known limitations.
• Audit log: Every model decision (approved/denied + model version + feature values at decision time) must be stored for 7 years per FCRA requirements.
• Lineage record: Every promoted model must have a registered lineage record: training script commit, data snapshot date range, preprocessing pipeline version, training run ID, gate results.
• Adverse action letter template: The model must be integrated with the adverse action letter generation system. The top contributing negative features must be translatable into the ECOA-required "principal reasons for adverse action."

Part B — Highest-risk gap: No audit log and no model lineage

The current system stores a pickle file on a serving server. There is no record of: which training data produced it, when it was deployed, what its predictions were, or which model version made each decision.

The failure scenario: A denied applicant files a regulatory complaint citing ECOA. The regulator requests all model decisions made for applicants with similar profiles over the past 18 months, along with the model versions that made those decisions and the feature values at decision time.

The current system cannot produce this. The pickle file does not record when it was deployed. The notebook run that produced it is not tracked. The individual prediction logs do not exist.

The regulatory penalty for failing to produce adverse action audit records can reach $10,000 per violation under FCRA. At loan application volumes of thousands per month, this is a material compliance risk.

Minimum viable fix: Before the next model deployment, add two components:

1. Prediction logging: Every scoring call logs: timestamp, application ID, model version ID, feature values, model output (score and decision), adverse action reasons if denied. Stored in an append-only data store with 7-year retention.
2. Model lineage tagging: The model serving endpoint tags every prediction with the model version ID. The model registry maps each version ID to its training metadata.

This can be implemented in 2–3 engineer-days and transforms a material compliance risk into a resolved operational requirement.

Part C — Maturity level and transition plan:

Current level: 0 (Manual)

Evidence: notebook-driven retraining, manual deployment by file copy, no quality gates, no monitoring, quarterly cadence.

Target level: 1.5 (Pipeline automation with compliance controls)

Full Level 2 CI/CD is not yet justified — the team is small, the model count is low, and the compliance requirements need to be stabilized before automation adds velocity. The target is a robust Level 1 with compliance-specific additions (audit logging, fairness gates, lineage tracking).

90-day transition plan:

Milestone 1 (Days 1–30): Foundation — reproducibility and compliance

• Implement prediction logging and model lineage tagging (compliance minimum viable)
• Convert the notebook to a parametric training script with a documented entrypoint
• Implement the model registry: every trained model stored with training metadata
• Add the three quality gates (metric floor, fairness check, adverse action coverage)

Milestone 2 (Days 31–60): Automation — pipeline and promotion

• Build the training pipeline as an orchestrated job (Prefect, Airflow, or cloud-managed equivalent)
• Implement automated quality gate execution on every training run
• Implement the human approval workflow: gate results surfaced in a dashboard, one-day approval window
• Implement rollback: the ML engineer can revert to the previous registered model in < 10 minutes

Milestone 3 (Days 61–90): Monitoring — detect before business impact

• Implement feature distribution monitoring with PSI alerts
• Implement the four monitoring signals (system, data, model, compliance)
• Define and document retraining triggers based on PSI thresholds
• Conduct a first monitoring review to baseline the alert thresholds

Self-assessment checklist:

• Did you specify a compliance-specific quality gate (not just a performance metric gate)?
• Did you choose manual promotion with automated pre-checks rather than fully automated or fully manual?
• Did your highest-risk gap answer identify the audit log absence as a regulatory exposure, not just a technical gap?
• Did your transition plan sequence compliance fixes before automation (not the other way around)?

Part A — Root cause diagnosis:

Code (pipeline change): The addition of "days since last purchase" is the most likely proximate cause. This feature — time since last purchase — creates a strong signal for user engagement recency. A model with this feature learns to recommend products to users who purchased recently (high recency = high engagement probability). The problem: recently purchasing users have lower incremental revenue potential. The model learned to optimize for users who were already going to buy, rather than converting users who were not. This is a proxy alignment failure operationalized through a feature addition.

Data: The feature change itself is the data-side contribution. The feature is informative — it improves the model's ability to predict whether a click will occur. The model is now better at predicting clicks. It is worse at predicting revenue-generating behavior, because click propensity and purchase propensity are correlated but not equivalent.

Model: The model improved at the proxy task (predicting clicks) and degraded at the business task (predicting revenue-generating purchases). This is expected — the model optimizes for what it is trained on. The AUC improvement of +0.04 reflects better click prediction, not better revenue prediction. The model did exactly what it was designed to do.

Root cause summary: The code change (feature addition) shifted the model's learned representation toward click-optimized behavior. The model now correctly ranks users by click likelihood. Users with high click likelihood in the "days since last purchase" dimension are recent purchasers who are browsing, not converting. CTR goes up because engagement goes up. RPU7 goes down because the system is now optimizing for browser engagement, not purchase conversion.

Part B — The upstream framing failure:

The recommendation system is using click-through rate as its model metric and proxy label. Chapter 1.1's proxy label rubric identifies click as a classic high-gaming-risk proxy for purchase intent: "Many carts never convert." The addition of a recency feature amplified this misalignment.

The upstream framing failure is that the product outcome was not clearly specified. "Recommend products" is not a product outcome — "increase purchase conversion among browsing users" is. If the product outcome had been specified as purchase-intent-driven, the proxy label would have been "add-to-cart + purchase within 7 days," not "click."

The feature addition made a misaligned proxy more powerful. Fixing the feature pipeline is not the root cause fix — aligning the proxy label to the business outcome is.

Part C — Operational changes that would have caught this:

Stage 4 (Validate and Register) — Multi-metric quality gate:

The quality gate checks AUC improvement. A multi-metric gate would have also checked the model's RPU-correlated metric: "predicted purchase probability" or "predicted add-to-cart rate" on a held-out set with known purchase outcomes. If CTR-AUC improves while purchase-AUC stays flat or declines, the gate should surface this divergence as a warning.

Specific gate: "New model must show AUC improvement on purchase label (add-to-cart + 7-day purchase), not only on click label. If click-AUC improves and purchase-AUC does not, flag for human review before promotion."

Stage 5 (Deploy) — A/B test design includes RPU as a primary metric:

The A/B test measured CTR in Week 3. RPU7 data arrived in Week 4 — after the ramp had begun. The A/B test should have been designed with RPU7 as the primary success metric and CTR as a secondary (guardrail) metric. Ramp decisions should be gated on the primary metric, not the secondary.

Specific operational change: "Model promotions for the recommendation system require A/B test results on RPU7 (7-day revenue per user) before any ramp above 10% traffic. CTR is a guardrail, not a decision metric."

Stage 7 (Monitor) — Business KPI proxy monitoring during ramp:

If the A/B test must run for multiple weeks before RPU7 results arrive, add a leading indicator: "add-to-cart rate per user" as an intermediate signal. Add-to-cart is available in hours; it is more aligned with purchase than raw CTR. A decline in add-to-cart rate during the ramp would have been an early warning before RPU7 results confirmed the problem.

Self-assessment checklist:

• Did you identify "days since last purchase" specifically as the feature contributing to the CTR/RPU divergence?
• Did you connect the root cause to proxy label misalignment (Chapter 1.1 framing failure), not just the feature change?
• Did your operational changes address Stage 4 (multi-metric quality gate), not just Stage 5 (better A/B test design)?
• Did you propose a leading indicator for the monitoring gap, not just "wait for RPU7"?

Part A — CI pipeline gates:

Gate 1: Code linting and type checking (on PR)

• What it checks: Python type annotations, import hygiene, code style (black/flake8), no hardcoded PHI (patient health information) in code
• Deterministic: yes
• Pass/fail: any linting error or PHI pattern match = fail

Gate 2: Unit tests for feature transformations (on PR)

• What it checks: Each feature computation function is unit tested against synthetic patient records. Tests verify: correct handling of null values (common in EHR data), correct date arithmetic for "days since last admission," correct encoding of categorical fields
• Deterministic: yes
• Pass/fail: < 100% test pass rate = fail. EHR-derived features have many edge cases; unit tests must be comprehensive

Gate 3: Schema validation on training data pipeline (on merge to main)

• What it checks: The training data pipeline output matches the expected feature schema — correct column names, data types, null rates within expected bounds (e.g., "age" null rate < 1%), and value ranges within clinically plausible bounds (e.g., heart rate between 20 and 300)
• Deterministic: yes (schema + statistical bounds checks)
• Pass/fail: any schema mismatch or out-of-bounds rate exceeds threshold = fail

Gate 4: Training sanity check (on merge to main, full pipeline run)

• What it checks: The model trains to completion; the final validation AUC is above a floor of 0.70 (minimum acceptable for clinical use); the top 3 risk factors produced for 100% of patients (coverage check); no PHI leaks into model artifact or prediction outputs
• Mix of deterministic and statistical
• Pass/fail: AUC < 0.70 = fail; top factor coverage < 99.5% = fail; PHI pattern detected in outputs = fail (automated blocker)

Gate 5: HIPAA compliance check (on merge, before any deployment)

• What it checks: Model does not directly use race/ethnicity features; prediction logging strips PHI before storage; audit log encryption verified; data retention policy enforced (7-year minimum)
• Deterministic: yes
• Pass/fail: Any HIPAA compliance check failure = hard fail, requires human review before proceeding

Part B — CT trigger strategy:

Trigger 1: Label batch arrival (every 30 days)

Every 30 days, a new batch of 30-day outcome labels becomes available. This is the primary trigger. When a new label batch arrives:

• Compute rolling AUC on the most recent 30 days of predictions (now labeled)
• If rolling AUC < current production model's validation AUC − 0.02 (meaningful degradation): trigger retraining
• If rolling AUC is within tolerance: log the metric, do not retrain

This is the "performance-based trigger" — it retains the model when it's still performing and retrains when measurable degradation appears. The 30-day label lag means this trigger fires approximately monthly.

Trigger 2: Feature distribution shift (weekly monitoring)

Weekly, run PSI on all input features for the past week's patients vs. the training data distribution. If any feature's PSI > 0.1 (moderate shift) or any feature's PSI > 0.2 (significant shift):

• PSI > 0.1: flag for review, no automatic retraining
• PSI > 0.2: trigger an early retraining review (the distribution has shifted meaningfully; wait for the next label batch to arrive before committing to retraining, but front-load the data collection)

Justification: Patient population characteristics can shift due to seasonal illness patterns, referral source changes, or hospital policy changes. Distribution shift is an early warning signal, not an automatic trigger — because the model may still perform well despite distribution shift if the relationship between features and outcome is stable.

Trigger 3: Volume anomaly

If daily prediction volume drops below 100 (50% of normal) for 3 consecutive days, alert the ML engineer. This may indicate an EHR data pipeline failure, not a model quality issue — but it produces stale predictions and must be investigated.

What should NOT trigger retraining:

• Calendar schedule alone: Monthly retraining on a fixed calendar date is wasteful if the model is performing well and the distribution is stable. Resource-constrained teams should retrain on evidence, not schedule.
• A single outlier day: A single day with unusual PSI should not trigger retraining — it may be a data ingestion artifact. Require 3 consecutive days of elevated PSI before escalating.
• Pressure from stakeholders: "The care coordinators feel like the model isn't as good" is not a retraining trigger. It is a signal to pull monitoring data and investigate. If monitoring shows no degradation, the model is not the problem.

Handling the 30-day label lag:

The label lag means the model is always training on data that is 30 days old. This is unavoidable given the outcome definition. Mitigations:

1. Use early indicators as proxy features: "emergency department visit within 7 days" is available in 7 days; "readmission for any cause within 7 days" is available in 7 days. These can be added as additional monitoring signals even if they are not the primary label.

2. The validation set for quality gates uses held-out labeled data (patients from 31–60 days ago, now fully labeled). The gate confirms the retrained model performs on data similar to the recent production population.

3. Accept the lag explicitly: document that the CT system responds to degradation with a 30-day minimum lag from onset to detection to retraining. Design the quality gate to ensure this lag does not allow a significantly degraded model to remain in production undetected.

Quality gate before promotion:

1. AUC on held-out labeled validation set ≥ 0.75 (clinical efficacy minimum)
2. AUC ≥ current production model AUC − 0.01 (no meaningful regression allowed)
3. Top-3 factor coverage ≥ 99.5% (care coordinators must receive reasons)
4. Fairness check: approval rates (high-risk classification) across age deciles and proxy indicators must not deviate > 10% (HIPAA-compliant equity check)
5. Adverse event rate on the 30-day labeled set: readmission prediction sensitivity ≥ 0.70 for high-risk patients (clinical safety gate)

Part C — Monitoring dashboards:

ML Team Dashboard (technical):

Care Coordination Team Dashboard (clinical/operational):

The care coordination dashboard does not show AUC or PSI — those metrics require ML expertise to interpret. It shows operationally actionable signals: is the system working, how many patients need outreach, and is the outreach completing on time.

Self-assessment checklist:

• Did your CI gates include a HIPAA compliance check as a hard blocker?
• Did your CT trigger strategy use performance-based triggers (not calendar alone) and explicitly address the 30-day label lag?
• Did your quality gate for promotion include a clinical safety gate (sensitivity on high-risk patients), not just AUC?
• Did you design separate dashboards for ML team vs care coordination team, with audience-appropriate signals for each?

1. The feedback loop causing the degradation:

This is distribution shift caused by the model's own predictions — sometimes called a closed-loop feedback effect or exposure bias in recommendation systems. The CT loop creates a self-reinforcing cycle:

• The model recommends popular content
• Users click popular content (because it was recommended, not necessarily because it's best)
• The next retraining dataset is dominated by clicks on popular content
• The new model learns even more strongly that popular content is what users want
• Diversity collapses further

In the bandit/recommendation literature, this is related to filter bubble formation through model self-training. The model has no counterfactual data — it has no signal about what users would have clicked if they had been shown different content. It optimizes on the interaction data it generates, which reflects its own past recommendations.

2. Breaking the cycle: counterfactual data injection

The most direct fix is exploration injection: in production, serve a small fraction (5–10%) of traffic with random or diverse content recommendations (epsilon-greedy or Thompson Sampling exploration). These "exploration" impressions and their resulting clicks are logged separately and included in the retraining dataset.

This provides counterfactual data: the model learns what users click on content it would not have otherwise recommended. Over time, the training distribution reflects actual user preferences across the content catalog, not just the preferences the model reinforced.

An alternative: inverse propensity score (IPS) weighting in the training loss. Downweight training examples from content that the model assigned high probability to (these are over-represented in the training data); upweight examples from content the model assigned low probability to. This corrects for the self-reinforcing bias without requiring explicit exploration traffic.

3. Diversity quality gate:

What it measures: Intra-list diversity (ILD) — the average pairwise content distance within each recommendation list. Concretely: for a list of 10 recommended items per user, compute the average embedding distance between all pairs. A diverse list has high average pairwise distance; a filter-bubble list has low average pairwise distance.

Operationally simpler proxy: catalog coverage at k — the fraction of the content catalog that appears in at least 1% of recommendation lists. A model that concentrates 70% of recommendations on 200 items has a catalog coverage of approximately 200/total_catalog.

Gate threshold:

• Catalog coverage must be ≥ baseline × 0.90 (the new model must cover at least 90% of the baseline's catalog coverage)
• If baseline catalog coverage was 8% (400 items recommended regularly), the new model must cover ≥ 7.2% (360 items)

Remediation path:

• If the diversity gate fails: do NOT auto-promote. Surface the failure with a comparison of the diversity metrics between the candidate and the current production model. Require human review before promotion.
• If the same model fails the diversity gate for 3 consecutive weeks: trigger a full review of the training data strategy. The retraining loop itself is generating training data that systematically degrades diversity — this is a structural problem, not a one-off.

4. CT trigger improvements:

Move away from fixed weekly retraining:

The fixed weekly schedule means the model retrains even when data quality or distribution is stable and even when the previous retraining made things worse. Alternative triggers:

1. Quality-gated trigger: Only retrain when the rolling 7-day AUC (on a held-out set with labels) falls below a threshold. If the model is still performing well, retraining is unnecessary and risks introducing the feedback loop amplification.

2. Diversity-triggered retraining: If catalog coverage metric drops below a threshold (e.g., 80% of baseline), trigger a special retraining run with exploration-augmented data, not the standard loop.

3. Exploration-first trigger: Before each scheduled retraining, verify that the training dataset contains at least 10% exploration traffic impressions. If the exploration fraction is below 10%, the CT loop is blocked until sufficient exploration data accumulates. This ensures the feedback loop is structurally broken before the model retrains.

Code: Can change without intentional deployment — through dependency updates in container rebuilds, base image updates, library version upgrades that occur as part of scheduled security patching. Check: container build timestamps for the serving environment in Q3. Check: Python and scikit-learn (or equivalent) version in the Q3 container vs Q2. If a serving library updated its feature computation (e.g., a change in how a datetime feature is computed, a change in how NaN values are handled in a scaling step), the feature values the model receives at serving time may have shifted without any explicit code change.

Specific evidence: compare serving container build dates to the timing of the false positive rate increase. Pull a sample of Q3 mobile new user feature vectors and compare the distribution to Q2 feature vectors for the same cohort.

Data: Most likely root cause. "New user mobile accounts" is a segment that can change composition dramatically without any pipeline change. In Q3:

• New mobile user acquisition channels may have launched (paid mobile ads, partnership referrals, new market expansion)
• New device types or operating system versions may be generating different behavioral signals
• New users from these channels have different feature profiles than the Q1 training population

The feature pipeline is unchanged — it still computes the same features. But the mobile new user population flowing through it has shifted. The model was trained on Q1 mobile new users. Q3 mobile new users have a different distribution of device age, session length, behavioral velocity, and transaction patterns. The model's learned boundary for "mobile new user fraud risk" no longer matches the actual distribution.

Specific evidence: Pull feature distributions for mobile new user accounts in Q3 vs Q1 (training period). Calculate PSI for each feature in this segment. If PSI > 0.1 on key features, the distribution has shifted.

Model: The model has not been retrained — but its effective performance can degrade without any change to the model artifact itself if the feature inputs change (above). The model is unchanged; its predictions are no longer calibrated for the Q3 mobile new user population.

Specific evidence: The model's predicted probability scores for mobile new user accounts in Q3 vs Q1 — has the score distribution shifted? Are Q3 mobile new users systematically receiving lower fraud probabilities (leading to more false negatives) or higher probabilities (leading to more false positives)? Given the symptom is increased false positives, the model may be assigning higher fraud scores to a new cohort of legitimate users.

Most likely root cause and fix:

Data distribution shift in the mobile new user segment is the most likely cause. The fix:

1. Immediate: adjust the decision threshold for the mobile new user segment specifically. A segment-specific threshold is a bandage, not a cure, but it reduces false positives while the root cause is addressed.
2. Structural: retrain with Q3 data that includes the new mobile user population. Add the mobile new user segment to the quality gate evaluation — the gate must check segment-level false positive rate, not just aggregate metrics.
3. Preventive: add segment-level distribution monitoring. The aggregate PSI was stable (Q2 and Q3 overall metrics are unchanged), but the mobile new user sub-segment drifted. Monitoring must be done at segment level, not just overall.

CI catches code, data, and training logic errors on every change before they reach production. CD safely releases pipeline and serving changes across environments. CT triggers retraining automatically when defined conditions are met. CM detects model and data quality degradation early and routes to corrective action.

Order matters because each loop depends on the previous one being stable. You cannot automate deployment (CD) safely until you can reliably test what you're deploying (CI). You cannot automate retraining (CT) safely until you can reliably deploy the retrained model (CD). You cannot act on monitoring signals (CM) reliably until the retraining and deployment they trigger are automated (CT + CD). Teams that try to build CT before CI end up automating retraining on untested pipelines — they accelerate the propagation of bugs, not the improvement of models.

Training-serving skew: the features the model receives at serving time are computed differently from the features used during training, causing the model to underperform in production despite strong offline metrics.

Causes: (1) Feature logic is implemented twice — once in the training pipeline (Python/Spark) and once in the serving layer (Java/C++) — and the implementations diverge. (2) Training data is joined with features from a point-in-time snapshot; serving computes features in real time, and the real-time computation differs slightly. (3) A feature available in the offline training dataset is unavailable, delayed, or approximate at serving time. (4) A data preprocessing step (scaling, encoding) is applied in training but missing or different at serving.

Detection: Log production feature values and compare their distribution to the training set distribution using PSI or KS test. If feature distributions differ significantly, skew is present. More directly: shadow the serving pipeline by logging raw inputs, running them through both the training pipeline and the serving pipeline, and comparing outputs.

Prevention: Share the feature computation logic between training and serving — a single codebase, not two implementations. Feature store: offline features for training and online features for serving computed by the same transform logic. Integration tests: run the same input through both pipelines and assert the outputs are identical within floating-point tolerance.

Both can be truthful and the quality can still have dropped — this is the three-moving-parts problem. "Same training data" typically means the same pipeline ran — not that the data flowing through the pipeline produced the same output. "Same model code" means the training script is unchanged — not that the trained model is identical to last week's.

Investigate in this order:

1. Compare the trained artifacts directly. Are the model weights meaningfully different from last week's? If yes, something changed during training even if the code is "the same." Check: random seed pinned? Library version identical? If the random seed was not pinned, the model trained to a different local optimum.

2. Compare training data feature distributions. Pull the actual feature values from last week's training run vs this week's. The pipeline code may be identical, but the data it processed may have different distributions if the upstream data source changed. Calculate PSI on each feature.

3. Check the environment. Did any dependency update between last week and this week? A scikit-learn patch release that changes default behavior in a transformer, or a numpy update that changes floating-point computation, can produce different model outputs from identical code and data.

4. Check the validation set. Did the held-out validation set change between runs? If the validation data distribution shifted, an 8% drop in online metrics with stable offline metrics suggests the offline evaluation is no longer representative of production.

The most common root cause for "same code, same data, worse model": an unpinned dependency updated between training runs, or the training data time window shifted and included a different distribution than the previous week's window.

A model registry is a versioned catalog of trained model artifacts — a source of truth for what is or has been in production, what produced each artifact, and whether each artifact has been validated.

Required metadata per registered model:

• Artifact path and hash (SHA256 of the artifact file — proves the artifact has not been modified)
• Training code commit (git hash or equivalent)
• Training data description: date range, row count, data source version
• Feature definition version: which version of feature transform code produced the training features
• Hyperparameters used
• Validation metrics: the full evaluation results including per-segment metrics
• Quality gate results: which gates passed and which failed
• Environment: Python version, key library versions used in training
• Lineage: which pipeline run produced this artifact
• Stage transitions: Candidate → Staging → Production → Archived, with timestamps and approver

Who reads from it:

• Deployment system: reads to determine which model version to promote to serving; reads the artifact path and hash to verify integrity before deployment
• Monitoring system: reads to correlate performance degradation with the model version currently serving; reads to know when to compare current metrics against the registered baseline metrics
• Audit/compliance team: reads to produce evidence of which model version made decisions on which dates; reads model cards for regulatory review
• Data scientists and ML engineers: reads to compare candidate models against the current production model; reads to understand what data and code produced any historical model

Level 2 (full CI/CD for pipelines and models) is justified when Level 1 operations create measurable bottlenecks that cannot be resolved by improving Level 1 execution.

Required evidence before recommending the transition:

1. Level 1 gates are trusted. Automated quality gates at Level 1 have been running for at least 4–6 weeks and have a demonstrated track record: they catch real issues without excessive false positives. If the team does not trust the gates, automating deployment based on them adds risk.

2. Coordination overhead is measurable and material. Each model promotion requires more than 2 hours of engineer time (including coordination, review, and manual steps). With 5+ models retraining weekly, this creates 10+ hours of overhead per week. This is the business case for automation.

3. Rollback is fast and tested. Before automating deployment, verify that rollback to the previous production model takes < 10 minutes and has been tested at least twice. Automated promotion without fast rollback is dangerous.

4. Incident rate is stable. The team has not had more than one production incident from a model promotion in the past 60 days. Automating promotions on an unstable system amplifies instability.

5. The team has a dedicated platform resource. Level 2 systems require ongoing maintenance — pipeline CI, deployment automation, quality gate calibration. Without at least 0.5 FTE of platform-oriented engineering capacity, the automation will decay faster than it adds value.

Without all five conditions, the better investment is typically strengthening Level 1 discipline rather than advancing to Level 2 tooling.

Scenario 1: Automated promotion gates that nobody calibrated. A team builds a Level 2 CI/CD pipeline with automated quality gates. The gates are set at arbitrary thresholds (e.g., "AUC ≥ 0.75") that were not derived from analysis of the business impact of different quality levels. The gate passes models that are technically above threshold but are subtly wrong in ways the threshold does not capture — like a model with 0.76 AUC that has 30% recall on the highest-stakes segment. The automation promotes this model; the business impact is not visible until a downstream incident.

Scenario 2: Continuous Training without monitoring. A team sets up a CT loop that retrains every week. They do not have CM (Continuous Monitoring) in place to tell them when retraining is needed or whether the new model is actually better in production. The CT loop runs regardless of whether the model is degrading. In a week where the training data contains a corrupted batch, the loop retrains on the corruption and automatically promotes a worse model. The feedback from monitoring that would have blocked this promotion does not exist.

Scenario 3: Feature store without feature ownership. A team deploys a feature store and adds dozens of features from multiple teams. Nobody assigns ownership to individual features — nobody is responsible for monitoring freshness, detecting staleness, or deprecating unused features. Six months in, the feature store contains features that are no longer being updated (stale), features with no documented meaning, and features that overlap with each other in undocumented ways. The tool is Level 2; the data governance practice is Level 0.

In all three cases, the team has the infrastructure of a higher maturity level without the operational discipline to use it safely. The infrastructure does not create the discipline — it amplifies whatever practices the team already has.

An ADR is a short document recording a significant architectural decision: the context that forced the decision, the decision made, the consequences (trade-offs and operational impact), and the alternatives that were considered.

ADRs solve the "why does it work this way?" problem. Without them, architectural decisions exist only in the memory of the engineers who made them. When those engineers leave, the decision is re-litigated from scratch — often making the same choice after the same debate, or making a different choice without understanding why the original was made.

Concrete MLOps example: "Decision to use batch serving instead of online serving for the churn model"

Context: The churn prediction model needs to serve scores for ~500,000 users daily. Initial design assumed online serving (REST API endpoint). The team discovered that 95% of use cases (email campaigns, account manager dashboards) need scores once per day in aggregate, not in real time. The remaining 5% (in-app retention offer) needs a score within 200ms.

Decision: Use batch serving as the primary serving mode. The training pipeline runs overnight and deposits scores to a database. The email and dashboard use cases read from the database. For the in-app use case, implement a score cache populated by the batch job — the in-app call reads from cache, not from a live model endpoint.

Consequences: Removes the need for a model serving endpoint and its associated SLO/latency/infrastructure overhead. Introduces a 24-hour staleness constraint — the in-app retention offer uses yesterday's score. If a user's churn risk changes dramatically intraday, the in-app offer may be based on stale information. Adds the cache invalidation responsibility.

Alternatives considered: (a) Online serving for all use cases — rejected because it requires a serving infrastructure (autoscaling, latency SLOs, redundancy) that is disproportionate to the marginal benefit of real-time scores for 95% of use cases; (b) Online serving only for the in-app use case with batch for others — rejected because maintaining two serving paths doubles the operational surface area.

They are not contradictory — they measure different things. The apparent contradiction is the classic symptom of a misaligned framing stack (Chapter 1.1). Understanding both measurements together is more valuable than choosing one to trust.

AUC (offline model metric): Measures how well the model ranks examples in the held-out validation set — relative ordering of predicted scores. Improved AUC means the model is better at distinguishing positive from negative examples in the historical dataset it was evaluated on. It does not prove the validation set is representative of current production data, it does not prove the model's improvement translates to the specific product interaction being measured (clicks), and it does not prove the decision policy (threshold, ranking mechanism) is capturing the model improvement.

Click-through rate (business/product metric): Measures a downstream product behavior, not model quality directly. CTR could be unchanged because: (a) the model improvement is not large enough to move CTR given the noise in the product, (b) the decision policy (ranking, threshold) is not surfacing the model's improved rankings to users, (c) other product factors dominate CTR and swamp the model signal, (d) the AUC improvement was on a segment or distribution not representative of the users who determine CTR.

Which to trust: Trust both, but for different purposes. AUC tells you the model is objectively better at its prediction task. CTR tells you whether that improvement translated to product impact. The question to investigate is why the AUC improvement did not move CTR — the answer usually reveals something about the decision policy, the measurement window, or the alignment between the model objective and the business metric.

The three models:

1. Separate DS + Platform/Ops: Data scientists and platform/infrastructure engineers are distinct teams. Platform builds and maintains the ML infrastructure; DS/MLE builds models on top of it. Clean ownership, but slow handoffs and friction in debugging.

2. Full-stack DS/MLE: Each data scientist owns their model from development to production. Fast iteration, tight feedback loops. Brittle at scale — depends on individual DS having full-stack skills; operational responsibility fragments across the team.

3. Hybrid platform-enabled: A small platform team (1–2 engineers) maintains shared infrastructure and golden paths. DS/MLE use the platform for standard workflows, with escape hatches for non-standard needs. Platform team's mandate: make the right path the easy path.

Recommendation for this team: Begin transitioning to hybrid platform-enabled now.

Current state (8 DS, 1 MLE, 3 models) is at the edge of sustainable full-stack operation. The 1 ML engineer is likely a single point of failure for all three production models. With 10 models in 12 months, the full-stack approach will break — 10 models owned individually across 8 data scientists with one operational engineer creates unmanageable coordination overhead and inconsistent practices.

The hybrid model does not require hiring a large platform team. The current ML engineer, with direction from the team, can begin building shared infrastructure components: a standardized training pipeline template, a shared model registry, a shared deployment mechanism. These become the "golden paths" that data scientists use by default.

Transition trigger: When the current ML engineer is spending > 50% of their time on operational overhead (deployments, debugging production issues, running notebook-based retraining for other data scientists) rather than building models, the team has exceeded the capacity of the full-stack model. At that point, platform investment has a clear business case.

A complete pre-production deployment checklist:

Reproducibility:

• Can this model be reconstructed from its lineage record (code commit + data snapshot + params + environment)?
• Is the artifact hash recorded and verified?

Quality gates:

• Has the model met the metric floor threshold on the held-out validation set?
• Has the model been evaluated on all important data segments (not just overall metrics)?
• Have fairness and compliance checks passed (if required by the domain)?
• Has latency been profiled at expected peak load? Does it meet the p99 SLO?

Deployment readiness:

• Is there a current production model to roll back to? Is that rollback tested and < 10 minutes?
• Is the serving infrastructure tested at the expected load?
• Are the monitoring signals (system, data, model, business KPI) in place and alerting correctly?

Operational ownership:

• Is there a named on-call engineer for this model?
• Does that engineer know how to execute a rollback?
• Is there a runbook for the most likely failure modes (serving error, quality degradation, data pipeline failure)?

Governance:

• Is prediction logging in place for audit purposes?
• Are the model card and lineage record complete and stored in the registry?
• Has the promotion been approved according to the documented promotion policy?

If any item is "no," the model should not be promoted until the gap is resolved. The checklist is not bureaucracy — each item corresponds to a documented production failure mode.