Credit score bias checks before you release a changed score

We ran this check on a public credit card dataset, and the pattern came from those accounts: we held sex, age, marital status and education out of the scorecard, then looked at approval rates by group at the cut-off. Each ratio came back near parity. If you own the score, it is tempting to take that as the answer and move on.

We did not stop at approvals. Splitting the same accounts into score bands showed younger borrowers with lower bad rates in every band. The score can meet a group-level approval comparison while risk still differs inside its bands. Keep the within-band bad rates alongside the cut-off results.

Approval parity is only the first check

Protected characteristics belong in the check, not in the score fit. In our public credit card dataset, we kept sex, age, marital status and education out of the scorecard and aside for validation. A characteristic can help you test the score for bias in algorithmic decisioning and data bias, and it stays out of the score.

Every group’s approval ratio was near one at the cut-off.

That is a useful first result, not a conclusion about how the score ranks risk. Teams can stop here because approval rates are easy to compare and look reassuring when they are close. But the cut-off turns a ranking into approvals and declines, and a group can still carry a different bad rate from other customers with the same score.

Compare bad rates inside each score band

The within-band bad rate is the check that shows whether groups in the same score band have similar outcomes. We split accounts into five score bands and compared each group’s bad rate with everyone else’s inside each band. A negative gap means the group defaulted less than others in the same band, so the score rated it riskier than its observed outcomes suggest. A positive gap means the score rated the group safer than its observed outcomes suggest.

Under-30s defaulted less than everyone else in every band. In the middle band, their bad rate was 13.5% against 15.9% for everyone else. The score also approved them slightly less often. The score therefore turns away some under-30s who repay better than it expects.

A gap that persists across the five random splits is more informative than one result from a single split. For the first four groups, the spread across splits was under one percentage point. The 60-and-over result varied too much across splits to interpret, so leave that gap out of your findings.

Compare bad rates within bands across multiple splits, including when the approval ratio is close or a band result is noisy.

Investigate the field behind a persistent gap

A persistent gap tells you where to investigate, not which field to remove. Among the score’s fields, the credit limit moved most with being under 30. The limit is the amount the issuer chose to give each customer, and under-30s had about half the median limit of everyone else. The score read a low limit as a sign of risk.

This is where a predictive field can carry the lender’s own earlier decision into the score. It also records what the issuer had already decided about the customer.

The women’s and married customers’ gaps had no field in this data that explained them. It is tempting to force an explanation because a gap without a cause is harder to act on. Keep the gap visible and assign a cause only when the evidence supports one.

For each group with a persistent gap, identify the field most associated with that group and check whether it records an earlier lender decision.

Put these checks into a decision flow

Create rules for credit, fraud or collections, then run them on live decisions.

Start free See pricing

Check every group after changing a field

A change that narrows one group’s gap must be checked against every group again. We replaced the credit limit with utilisation, the last bill divided by the limit. Utilisation retains information about how stretched the customer is but removes the amount the issuer chose to give.

Each value in both columns is the average gap across the five score bands, in percentage points.

Under-30s had a smaller gap after the change, but the graduate-school gap widened. Women’s result stayed the same, and the married group’s gap narrowed slightly. The limit was also associated with education, so removing it changed what the score said about graduate-school customers.

The Gini was virtually unchanged, and the approved book’s bad rate remained 13.1%. The overall score held its predictive power, and its errors shifted across groups.

The band-level comparison matters because an average gap can hide what happens within the score ranges where your cut-off operates.

The under-30s defaulted less in every band when scored with the limit. The change narrowed their gap, but it did not resolve every group’s result. Rerun the band check for every group after each change.

Backtest the changed score before release

A backtest shows which past decisions the changed score would alter. It runs a release against decisions already made and reports changed approvals and declines, with realized outcomes where the history has them. Those changes affect who enters the book, so the past outcomes help you assess the consequences before the score goes live.

Run the changed score against past decisions and review the changed approvals and declines, including their realized outcomes where available, before release.

Run the full check on your score

If you own a score whose group approval ratios are near one, this is the check to run next.

1. Keep protected characteristics out of the score fit and available in separate data for checking.
2. Run approval ratios by group. If every ratio is near one, continue the check.
3. Compare each group’s bad rate with everyone else’s inside each score band, across more than one split.
4. For a group with a persistent gap, find the field most associated with that group and ask whether it records an earlier decision by the lender.
5. Change the field, rebuild the score, and rerun the band check for every group.
6. Backtest the changed score on past decisions and review what would change before release.

Questions about checking credit score bias

How do I check a credit score for bias?

Keep protected characteristics out of the score fit, compare approval ratios by group, then compare each group’s bad rate with everyone else’s inside each score band across multiple splits. Investigate fields behind persistent gaps and rerun the group checks after a change.

Is removing protected attributes from a score enough?

No. A score can omit protected characteristics and still misjudge a group through other fields. Check outcomes inside score bands as well as approval rates at the cut-off.

Why can approval rates look equal while the score is off for a group?

Approval rates summarize who falls above the cut-off. Inside a score band, customers with similar scores can still have different bad rates, which reveals a mismatch that the cut-off comparison does not show.

What do I do about a gap with no field behind it?

Keep the gap visible and do not assign a cause or change a field without evidence. The check does not establish an acceptable-gap threshold.

How did the change affect the under-30 gap and predictive power?

It narrowed the under-30 gap from minus 1.5 to minus 0.5 percentage points, but did not close it. Gini was virtually unchanged, and the approved book’s bad rate remained 13.1%. The change did widen the graduate-school gap, so check every group again.

Put these checks into a decision flow

Create rules for credit, fraud or collections, then run them on live decisions.

Start free See pricing