An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

How Analytics Gets Builtadvanced

Getting analytics into production

Q: A churn model has run for a year. Outcome labels arrive 90 days late, so measured performance always describes the previous quarter. Which monitoring gives the earliest warning that the incoming population or the pipeline has changed?

The distribution of the model inputs and of its output scores, both of which can be checked on the day of each run. Accuracy against delayed labels is a lagging indicator by construction and cannot detect anything sooner than 90 days. Input and score distributions are observable on the day of the run, so they catch a shifted input population or a broken feed at once. Note what they cannot catch. Under pure concept drift the inputs can sit exactly where they always did while the relationship to the outcome moves, and neither distribution has to shift at all, so that failure surfaces only when the delayed labels arrive. Waiting for the business number to move is later still, and blind monthly retraining changes the model without diagnosing anything.

Q: A fraud model is retrained each month using only the transactions it allowed through. What is the risk?

The model learns from a population it filtered itself, so the transactions it refused never come back with an outcome that could correct it. Blocked transactions never complete, so no genuine outcome is ever observed for them and a customer refused in error is never contradicted. Fraud the model waves through is the opposite case: it settles, and a chargeback or an investigation can attach a label to it later, so those misses do reach the training data. The missing side is the blocked one, and retraining on allowed cases alone treats past refusals by the model as if they had been correct. Breaking the loop needs outcomes for refused cases, from a deliberate sample passed through for review or an independent check on a share of refusals.

A result becomes an asset only when it runs on a schedule that somebody depends on. Production means a defined input contract, a run that fails loudly rather than quietly, a named owner, a monitoring plan and an agreed retirement point. The model itself is a small part of the system, and the surrounding data plumbing, configuration and monitoring dominate the ongoing cost. Two things then go wrong over time. Data drift is a change in the inputs. Concept drift is a change in the relationship between inputs and outcome. Both degrade a model that was correct on the day it shipped.

Why it matters

Shipping a model is like handing over a car rather than a photograph of a car. Something is now running without you, in weather you did not test, and it will quietly get worse. Nobody notices a model decaying, because it keeps returning numbers with the same confident formatting. That is why monitoring is not optional housekeeping. It is the only thing standing between a working system and a silently wrong one.

Before you read on — recall

A churn model has run for a year. Outcome labels arrive 90 days late, so measured performance always describes the previous quarter. Which monitoring gives the earliest warning that the incoming population or the pipeline has changed?

Formulas

Two kinds of drift, separated

P(X, Y) = P(Y \mid X)\, P(X)

Data drift is movement in

P(X)

, the distribution of the inputs, while the mapping from inputs to outcome holds. Concept drift is movement in

P(Y \mid X)

, so the same inputs now imply a different outcome. The distinction matters because it points to different responses. Retraining on recent, representative labels can address concept drift, but only where the new relationship is still learnable from the features you hold. A changed objective or a changed set of drivers calls for different features or a different model. Retraining can help under input drift too, so diagnose which one you are facing before choosing the fix.

Population stability index for input drift

\mathrm{PSI} = \sum_{i=1}^{k} \left(a_i - e_i\right) \ln\!\left(\frac{a_i}{e_i}\right)

Bin a variable, record the expected share

e_i

in each bin from training data and the actual share

a_i

from recent data. With expected shares of 0.20, 0.50 and 0.30 and actual shares of 0.35, 0.45 and 0.20, the three terms are about 0.084, 0.005 and 0.041, giving a PSI near 0.13. A widely used credit-scoring convention treats values under 0.1 as stable, 0.1 to 0.25 as worth investigating and above 0.25 as a material shift.

Worked examples

Scenario

A demand model was trained when about 20 per cent of orders arrived online. Two years later that share is 55 per cent. Accuracy has fallen but no code has changed and no error has been raised.

Solution

This is data drift. The relationship between order channel, lead time and demand may be unchanged, but the model now spends most of its time in a region of the input space it barely saw in training, so it is extrapolating rather than interpolating. Input monitoring would have flagged it early: the channel variable's population stability index would have crossed 0.25 long before the business noticed. Diagnosis comes first, then the response. The options here are retraining on recent data that represents the current channel mix, refitting with the online segment properly weighted, or accepting the degradation against a documented tolerance. Re-binning cannot help, because a binary channel variable has no bins to redraw. The unacceptable option is not knowing.

Scenario

A churn model runs monthly. Outcome labels only become known 90 days after a prediction, so any measured accuracy describes the world a quarter ago.

Solution

Label delay makes performance monitoring a lagging indicator, so it cannot be the primary alarm. Two leading indicators are available immediately. The input distributions can be compared against training, which detects data drift on the day it happens. The distribution of the model's own output scores can be tracked, since a sudden shift in the share of high-risk scores signals either an input problem or a genuine change in the population. Performance against delayed labels still runs, but as confirmation rather than as detection.

Common mistakes

✗Production is a final deployment step tacked onto the end. What can run in production constrains what may be built. A feature that cannot be computed at the moment a decision is made is unavailable, no matter how predictive it looked during training.
✗A model validated once stays valid. Inputs move and relationships move. Without monitoring, degradation is invisible, because the system keeps producing well-formatted numbers right up until someone notices a business figure has drifted.
✗Drift means the model is wrong. Drift means the world has moved relative to the training data. The correct response is sometimes to retrain, sometimes to adjust the input representation, and sometimes to accept the change against a written tolerance. Diagnosis has to come before the fix.
✗Automatic retraining on recent data solves drift. It can entrench the problem. A fraud model retrained only on the cases it allowed through learns from a population that the model itself filtered, so its existing blind spots are reinforced rather than corrected.

Revision bullets

•Production means input contract, loud failure, named owner, monitoring, retirement plan
•The model is a small share of the total system and its ongoing cost
•Data drift moves the inputs; concept drift moves the input-to-outcome relationship
•Population stability index compares recent input shares against training shares
•Label delay makes performance a lagging alarm, so monitor inputs and scores too
•Training and serving must compute a feature identically or the model degrades silently

Quick check

A fraud model is retrained each month using only the transactions it allowed through. What is the risk?

Connected topics

Analytics Workflow Why Analytics Stalls ETL and ELT Cloud for Analytics Models & Abstraction Supervised vs Unsup.CRISP-DM Analytics Capability Agile Analytics Data Governance Ethics by Design Analytics as a Service Agentic AI

More in How Analytics Gets Built

Why Methodology Structuring a Problem Business Model Mapping Analytics Leverage Design Thinking

Sources

Sculley et al. (2015)
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems 28, 2015.
Shows that model code is a small fraction of a deployed system and names the surrounding maintenance costs.
Gama et al. (2014)
Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. "A Survey on Concept Drift Adaptation." ACM Computing Surveys, 46(4), Article 44, 2014.
The standard reference separating changes in the input distribution from changes in the input-to-outcome relationship.
Siddiqi (2017)
Siddiqi, N. Intelligent Credit Scoring: Building and Implementing Better Credit Risk Scorecards. 2nd ed. John Wiley & Sons, 2017.
Practitioner source for the population stability index and the rule-of-thumb thresholds quoted above.

How to cite this page

Dr. Phil's Quant Lab. (2026). Getting analytics into production. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-into-production