Model monitoring

A model is checked once, on the decisions it was built from. After that, the customers, the data feeding it and the way people use it keep changing. Model monitoring watches a live endpoint on a schedule and tells you when the decisions it makes no longer look like the ones you started from.

What a monitor does

One monitor watches one endpoint. You choose what to watch in the decisions it returned:

  • the score, if your decision output carries one,
  • the decision, such as approve or decline,
  • the inputs that matter most, up to 30.

You also choose a reference window, a stretch of past days that represents how things should look. Every later period, weekly or monthly, is compared with the decisions made in that window. The rows themselves stay where the endpoint archives them. Only counts are read.

A monitor answers one question: is the population still the one I built on? It does not say whether the model still ranks risk correctly, because that needs outcomes, which arrive months later. For performance on outcomes, see Score performance. A monitor is the early warning. Inputs change first, and performance follows.

A monitor reads the decisions the endpoint has archived, so the endpoint must have been receiving traffic during the reference window.

Create a monitor

Open Model monitoring from the menu, or from Analyses where it sits beside the other tools, and press New monitor.

FieldWhat to enter
NameA title for the monitor. If you leave it empty, the endpoint name is used.
Endpoint nameThe endpoint whose decisions are watched. Required.
Score in the decision outputThe path of the score in the decision output, for example $.score. Leave it empty if there is no score.
Decision in the decision outputThe path of the decision. It starts as $.decision.
Inputs to watch, one path per lineThe inputs to compare, for example $.applicant.income. Paths start with $. followed by field names. At most 30.
Reference from, Reference toThe reference window. Choose days when the model behaved the way you want to hold it to.
How oftenWeekly or Monthly.

At least one of score, decision or inputs must be named. Press Start monitoring. The monitor opens on its own page.

A few things to know when you choose the reference window:

  • It needs enough decisions to describe a normal period. If it holds too few, the monitor is refused and asks for a longer window.
  • It is frozen. Later periods never change it. If the model is retrained or the business changes for good, create a new monitor on a new window.
  • The first period is checked once the reference window has closed and one cadence has passed. With a reference that ends on 1 October and weekly checks, the first period is checked on 8 October.

Every period looks at the days just closed: the last 7 days for a weekly monitor, the last month for a monthly one.

Where a monitor starts

You can also start a monitor from a model that was trained by a policy pipeline. On the model page, next to the pipeline that trained it, press Monitor this model. The form opens with the name, the endpoint and the reference window taken from the pipeline, and with the model's inputs already listed. Check the score and decision paths, then press Start monitoring.

See Model for importing and using models, and Policy mining for pipelines.

The list

The list shows one row for each monitor.

ColumnWhat it shows
NameThe monitor. Select it to open its page.
Endpoint nameThe endpoint it watches.
Latest periodThe verdict of the last checked period, or the monitor's state if it has no verdict yet.
Last checkedWhen the last period was checked.

Before a monitor has a verdict, the badge shows its state:

StateMeaning
Reading the referenceThe monitor is learning from the reference window.
Waiting for the first periodThe reference is read. The first period has not been checked yet.
PausedYou paused it. No periods are checked until you resume.
Overdue: the last check did not runA check was due but did not run.
FailedThe monitor stopped on an error. Its page says why.

The trash icon on a row deletes the monitor after you confirm.

Read a monitor

A monitor page leads with the answer: a badge, and one sentence saying how many decisions were compared, and which dates.

The verdict

VerdictMeaning
StableNothing watched has moved enough to matter.
DriftingAt least one watched part is moving away from the reference. Look at it before it grows.
MovedAt least one watched part has moved a long way from the reference. Investigate.
Too few decisionsThe period holds too few decisions to judge. The page says how many it holds and how many are needed. Do not read this as stable.

The verdict is the worst result among everything watched. One input that has moved is enough to make the whole period Moved.

Under the badge, up to four reasons say what decided it, in plain sentences. For example, an input that moved away from the reference, an input that is missing in a larger share of decisions than before, values the reference never had, or values outside the reference range.

How far each part moved, period by period

A line chart shows the Score, the Decision and the Most-moved input for each checked period. The vertical axis is the PSI (population stability index). Two lines mark the points where a part counts as Drifting and as Moved. A line that climbs period after period is a slow change. A line that jumps is a sudden one.

Spread against the reference

Up to three bar charts compare the reference with the current period:

  • the score spread,
  • the decision mix,
  • the spread of the most-moved input.

The grey bars are the Reference and the blue bars are This period. Where the bars differ, the population has changed. Numeric values are grouped into ranges. Text values show each value, with Other values collecting the ones that are new.

The inputs table

The table lists each watched input, the most-moved first.

ColumnWhat it shows
InputThe field that is watched.
PSIHow far it moved. Higher means further.
VerdictStable, Drifting or Moved, by the same limits as above.
Missing, reference → nowThe share of decisions where the input was empty, before and now.
Data qualityFlags for missing values, values outside the reference range, or values the reference never had. A dash means nothing was flagged.

Data quality problems often explain a move: a field that stopped arriving from a source, a unit that changed, or a new category from a changed form. Check the data feeding the endpoint before you judge the model.

How this is checked

At the bottom, How this is checked opens a short note with the reference window and how many decisions it holds, the limits for drifting and moved, and the cadence and endpoint.

How the verdict is reached

For the score, the decision and each input, the monitor splits the values into groups and compares the share of decisions in each group with the reference. The comparison is the PSI.

PSIResult
Below 0.1Stable
0.1 to below 0.25Drifting
0.25 or moreMoved

Numbers are grouped by ranges taken from the reference window, and anything beyond them is counted as out of range. Text values are compared as they are, with values the reference never saw counted together.

Alongside the PSI, three data quality checks run on every input. Each is flagged on its own:

  • the share of decisions where the input is missing rose by 5 points or more over the reference,
  • 2% or more of numeric values fall outside the range of the reference,
  • 2% or more of text values were never seen in the reference.

A period needs at least 200 decisions before it is judged. Below that, the result is Too few decisions.

Pause, rename and delete

On a monitor page, Pause stops the checks and Resume starts them again. A paused monitor keeps its history. Use it while an endpoint is being rebuilt or tested and its decisions would not be representative.

Select the title, or use the settings menu and choose Options, to rename the monitor. The endpoint, the paths and the reference window cannot be edited. To change those, create a new monitor.

To remove a monitor, delete it from the list.