A machine learning model goes into production making excellent predictions. For the first few months it is almost always right, the team grows confident, and decisions start to lean on its output. A year later, without anyone touching a single line of code, the predictions fail more often than they should. The most dangerous part is that almost no one notices: there are no errors on screen, no exceptions in the logs. The model keeps answering — it just answers worse.
This phenomenon has a name: model drift. It is one of the most underestimated reasons an AI project stops delivering value, and it is treacherous precisely because it is silent. Unlike a service that goes down, a drifting model does not shout: it degrades bit by bit, and the bill shows up in the wrong decisions that pile up without any alarm.
Understanding what drift is, why it happens and how to watch for it is what separates a team that ships models from a team that maintains them. Throughout this article we look at the types of drift, the signals to watch and a monitoring routine that does not require an expensive platform to get started.
What model drift actually is
A model learns a relationship between what it observes — the input variables — and what it is trying to predict. That relationship is estimated from historical data, capturing the world as it was during a specific period. The problem is that the world does not wait: customers change habits, prices rise, new competitors appear, seasonality shifts, the economy fluctuates. When reality moves away from what the model saw during training, prediction quality drops. That gap is what we call drift.
It helps to separate two concepts that are often confused. A bug is a defect in the code that consistently produces wrong behaviour. Drift is not a defect: the code is correct and the model is doing exactly what it was trained to do — the target simply moved. Fixing a bug is repair; dealing with drift is maintenance. They are different disciplines, and treating them as the same thing is the first mistake.
Why a model gets worse when nobody touched it
Anyone coming from traditional software carries a solid intuition: if the code does not change, the behaviour does not change. With models, that intuition fails, because the behaviour does not depend only on the code — it depends on the data flowing in with every prediction. A report that sums sales always returns the same value for the same data. A model that estimates the probability of a customer cancelling depends on behavioural patterns that do change over time.
For that reason, maintaining a model is neither optional nor a sign that it was badly built. It is a natural consequence of living in a dynamic world. The useful question is not whether a model will drift, but when and how fast — and whether the organisation will notice in time.
The types of drift: data, concept and upstream
Not all drift is the same, and telling the types apart helps you choose the right response.
Data drift happens when the distribution of the input variables changes. The model still receives the same fields, but their values shift: the average customer age rises, the average basket changes, one region starts to weigh more in sales. The relationship the model learned may still hold, but it now operates on terrain it barely knows.
Concept drift is deeper: the very relationship between inputs and outcome changes. What used to signal a good customer stops signalling it. A classic example is fraud detection: as fraudsters change tactics, the patterns that once exposed a fraud become harmless, and vice versa. Here, even if the input distributions do not change much, the model loses predictive power.
There is also upstream drift, often forgotten: changes in the data pipelines themselves. A unit that switches from euros to thousands of euros, a field that changes format, a new category that appears without warning. It is not the world that changed — it is the way the data reaches the model. It is the easiest cause to fix and, curiously, one of the most common.
Signs that your model is drifting
Drift rarely arrives with a clear warning, but it leaves a trail. It is worth watching systematically:
- A drop in performance metrics — prediction error rises (for example, the MAPE of a demand model) or classification accuracy falls, compared with the baseline period.
- A change in the input distributions — variables whose means, spreads or proportions differ from those at training time.
- A change in the prediction distribution — the model starts predicting far more (or far less) of one class than it used to.
- More complaints or manual corrections — when business teams start fixing by hand what the model suggests, it is an early and cheap signal.
- A gap between predicted and actual — whenever the real outcome eventually becomes known, comparing it with what was predicted is the most honest test.
How to monitor drift in practice
You do not need an expensive platform to start. The key is to choose a baseline period (typically the data the model was trained and validated on) and to compare, on a recurring basis, the new data against that baseline.
For input variables, a widely used metric is the Population Stability Index (PSI), which quantifies how far one distribution has moved from another:
PSI = Σ (perc_new − perc_baseline) × ln(perc_new / perc_baseline)
As a rule of thumb, a PSI below 0.1 suggests stability, between 0.1 and 0.2 a moderate change that deserves attention, and above 0.2 a significant change worth investigating. For performance, the ideal is to track the metric that matters to the business — MAPE, MAE, accuracy, recall, as the case may be — whenever the actual outcome becomes available. And, above all, define thresholds and alerts before there is a problem, so that drift is caught by a dashboard rather than by an unhappy customer.
When (and how) to retrain the model
Detecting drift is half the job; the other half is responding. There are two philosophies, and the best teams combine them. The first is scheduled retraining: updating the model at a fixed cadence (monthly, quarterly), regardless of whether there is drift. It is simple and predictable, but it can retrain too much or too little. The second is trigger-based retraining: updating only when monitoring shows performance has fallen below a threshold. It is more efficient, but it requires that monitoring to be working.
Retraining is not the only possible response, and it is not always the right one. If the cause is upstream drift, the correct move is to fix the pipeline, not retrain on corrupted data. If concept drift is abrupt — a regulatory change, an economic shock — you may need to rethink the variables, not just re-feed the same model. Retraining blindly on recent data can even lock in an undesirable pattern.
Common mistakes in managing drift
The most frequent mistake is not monitoring at all: shipping the model and assuming it is as good as it was on demo day. The second is confusing input stability with prediction quality — the distributions can hold and performance can still fall (concept drift). The third is retraining by reflex, without investigating the cause, which masks data problems instead of solving them. And the fourth, more organisational, is having no owner: a model with no one responsible for its health is a model that, sooner or later, no one realises has stopped being any good.
Mini case: when demand stopped obeying the model
A retail company used a demand forecasting model to plan purchasing and replenishment. For more than a year, the average error (MAPE) hovered around 8%, and the buying teams trusted the forecasts. Over a few months, a shift in buying behaviour — greater price sensitivity, against a backdrop of rising costs — changed the patterns the model had learned. With no change to the code, the MAPE rose to around 19%, and the model began to overestimate demand across several categories, generating excess stock of roughly 14%.
What saved the day was not luck, it was vigilance. A simple dashboard tracked the monthly MAPE and the distribution of two key variables; when the error crossed the defined threshold and the PSI of one of those variables passed 0.2, an alert fired. The team investigated, confirmed it was concept drift and not a pipeline problem, and retrained the model on recent data, also adjusting a price variable. The MAPE returned to near 10% and the excess stock normalised over the following months. Without monitoring, the same fix would have arrived months later — and much more expensive.
In practice
A model in production is not a finished product; it is a relationship with a world that changes. Model drift is how that world sends the bill, and the only realistic defence is to assume it will happen and prepare to catch it early. Start small: pick one performance metric, one or two critical variables, define thresholds and build a dashboard someone looks at regularly. Decide, from the outset, who owns the model's health and what the retraining policy is.
The good news is that managing drift does not require expensive tools to get going — it requires discipline and clarity about what you are watching. The organisations that treat model maintenance as part of the job, rather than an extra, are the ones that keep extracting value from AI long after the excitement of launch has faded.
