Scores and models

This playbook helps you judge an existing score, screen characteristics, check whether a model remains stable, or build a scorecard from records, decisions or events. It explains which evidence supports each answer and how to read the result before you use a scorecard in a decision flow. For data sources and mappings, see Analysis data. For starting an analysis, repeating it, email findings and opening cases, see Running an analysis.

Choose the analysis

Choose the question that matches the decision you need to make.

AnalysisUse it to answerWhat it returns
Score performanceDoes an existing score separate the outcome, remain stable between periods and produce honest probabilities?Performance measures, period and group comparisons, hypotheses and watch-outs
Characteristic strengthWhich characteristics are useful candidates before you build a scorecard?Information value and outcome rate by band for each characteristic you choose
Model stabilityHave the score, decision or selected inputs changed compared with a reference window?Stability measures and checks for missing, out-of-range or new values
Scorecard from an outcomeCan you fit a scorecard from one table of records and their outcome?A scorecard, validation results and a cutoff curve
Scorecard from decisionsCan you fit a scorecard from recorded decisions and their outcome?A scorecard based on the decision archive, validation results and a cutoff curve
Scorecard from records and eventsCan records and their payments over time support a scorecard and a decision flow?A scorecard, validation results, score bands and a decision flow when the card is strong enough
Score a new tapeWhat scores would a released decision flow assign to every record in a new file?Scored records and their distribution

Scorecard from decisions lives in Policy mining. The Scorecard page explains how to read a released card, and Binning and stability covers bands and stability in more detail.

Judge an existing score

Score performance

Score performance tests how well a score separates an outcome such as default, fraud or chargeback. It can compare periods and groups, and it can optionally test whether a second score separates better.

The analysis tests whether the score separates with Gini at least a floor. It also tests whether the largest stability index between periods stays within its limit. When the score represents a probability, it tests whether the predicted probabilities match the observed outcome rates with Hosmer-Lemeshow.

MeasureWhat it tells you
GiniHow well the score ranks records from lower to higher outcome risk.
AUCHow well the score ranks records by separating the outcome from the good result.
KSThe widest gap between the distributions of bad and good records.
Information valueHow much a variable separates records by outcome.
Stability indexHow far a population moved between periods or reference groups.
Hosmer-LemeshowWhether predicted probabilities are honest when compared with outcome rates.

Read the measures together. A score can rank records well while its population has moved between periods, or its probabilities can fail to match observed rates. A result that supports the hypotheses gives you evidence for continued use. A rejected hypothesis identifies the property that needs investigation. A second score that separates better gives you a comparison for deciding whether it deserves further testing.

The analysis checks for few outcomes, information value that is too high to be honest, and a Gini result that falls in a period or group. Treat a result affected by few outcomes as Directional, because the sample is too small for the number to be firm.

Characteristic strength

Characteristic strength reports information value and outcome rate by band for each characteristic you choose. Use it to screen candidates before building a scorecard. A characteristic with information value so high that it was probably recorded after the outcome is a watch-out. Exclude that characteristic from consideration for a scorecard because it would use information that was unavailable at the decision point.

Check model stability

Model stability compares a period with a reference window for the score, the decision and the inputs you select. It measures population movement and checks whether values are missing, outside their expected range or new to the reference.

The watch-outs name the change found:

  • Moved means the watched values changed from the reference.
  • Drifting means the watched values are gradually changing.
  • Missing more often means missing values increased.
  • Out of range more often means more values fall outside the reference range.
  • New values the reference never had means the period contains values absent from the reference.

The analysis also checks whether there are too few decisions for a firm verdict. Treat that result as Directional. A model rarely fails at once. Its inputs change first, so use these findings to investigate the data and the decision population before performance fails.

To run this comparison on a schedule against a live endpoint, set up a monitor. See Model monitoring.

Build a scorecard

A scorecard turns characteristics into points that a person can read and trace. Each characteristic is cut into bands. Each band receives points in proportion to how much worse or better its outcome rate is than average, using weight of evidence.

The characteristics are combined by a regularised logistic fit. Correlated characteristics share the credit instead of competing for the same effect. The points are scaled so that a fixed number of points doubles the odds. You can read that convention from the card.

This structure lets you trace each point to a band. A credit officer can read the reason for a score, and a regulator can audit how the score was formed.

Select the evidence

The three scorecard analyses use different evidence:

  • Scorecard from an outcome uses one table of records and their outcome.
  • Scorecard from decisions uses recorded decisions and their outcome in the decision archive.
  • Scorecard from records and events uses records and their payments over time and builds a decision flow.

For the records-and-events analysis, Compare within names a segment held as a control. Its effect is fitted so it does not distort the other characteristics, and it is excluded from the score. This prevents a segment that you price by from being counted twice. Hold out one at a time names the groups used for the holdout.

You can use challengers to fit alternative cards for comparison with the main card. Minimum support sets how many rows a band must hold.

Validate the card

Validation is the point of the analysis. The card is judged only on data it did not learn from.

For decisions and outcomes, the default split is by time. Earlier rows teach the card and later rows judge it, which catches change. You can use a random split instead, which catches noise.

For records, validation uses groups the card never saw, one group at a time. A random split can let the group effect leak between the learning and judging data and make the card look better than it is.

The score must separate on held-out rows, with AUC at least a floor. The gap between training and validation must stay small. A large gap means the card has memorised its learning data.

The analysis checks for too few rows, a rare outcome, a card that kept very few variables, very few payers and large differences between groups. A card that works very differently by group has a watch-out when the best group is much better than the weakest.

Require ordered bands

For a records-and-events scorecard, records are cut into five score bands by share. The bands are named from worst to best. The bands must rank in order of value on held-out data, and the analysis tests this as a hypothesis.

The analysis builds a decision flow from the card when the card is strong enough on groups it never saw. If the card is too weak on those groups, the run blocks and tells you why no decision flow was built.

Read the result

The result page separates evidence from interpretation.

Hypotheses contains statements the analysis tests. Each has a verdict of held, rejected or unclear, with the number behind it. Use held as evidence that the tested condition is supported. Investigate a rejected condition. Treat unclear as insufficient evidence, especially when the result is Directional.

Watch-outs contains known traps the analysis checks for. A watch-out stays quiet, warns or blocks the run. A warning requires investigation before you rely on the result. A block means the analysis has stopped because its evidence does not support the requested output.

Findings brings the important results together. A finding can be New, Still open or Cleared. Use the state to distinguish a newly raised issue from one that remains unresolved or has been addressed.

Steps is a collapsed detail view of each step the run took and what it returned. It is evidence of how the answer was reached. It is not advice and it holds no actions.

A completed scorecard result gives you the card with its bands and points, a cutoff curve and the validation numbers. It also gives you a button to open the scorecard as a revision that you can test and release. Nothing is deployed by the analysis. Use the validation evidence and the watch-outs before you test or release that revision.