Reject inference: test each decline rule before replacing it
We ran a lender-style test on a public credit-card dataset and FICO's public HELOC applications, both with known outcomes. The first step was deliberately familiar: apply one decline rule, then develop the scorecard on the applications that passed. In the card data, the rule rejected anyone behind on last month's payment. When we reviewed the resulting variables, the field that recorded that missed payment had an information value of 0.02, below the usual cut and easy to mark for deletion. The result did not look dramatic. It looked like routine housekeeping in a model-development workbook. But if your policy has already acted on the field, is its low information value evidence about credit risk, or evidence about selection?
What the methods predicted for the declined group
We compared the scorecard built on approved accounts with common reject inference methods. Each method starts with that scorecard, makes an assumption about the declined accounts, then uses those inferred outcomes to build another scorecard. Reject inference is an estimate of declined applicants' creditworthiness when their performance data is missing. The table shows what each approach predicted for the declined group, against the known bad rate in each dataset.
| Method | Card data | HELOC data |
|---|---|---|
| Real bad rate | 50.4% | 72.4% |
| All outcomes known | 50.5% | 72.5% |
| Approved only | 18.9% | 71.6% |
| Fuzzy augmentation | 20.3% | 71.8% |
| Hard cutoff | 80.7% | 90.1% |
| Parceling | 29.1% | 90.8% |
Fuzzy augmentation adds declined accounts twice, weighting each as both good and bad according to the first scorecard's predicted probability. It then feeds those predictions back into a rebuilt scorecard. Hard cutoff labels each declined account good or bad based on its predicted risk compared with the median approved account. Parceling divides declined accounts into score bands and draws bad outcomes at an increased rate based on the approved accounts in each band. These are assumptions applied to declined rows, not observed outcomes.
On the card data, none of the methods came close to the real bad rate. On HELOC, the approved-only scorecard was already close, and the methods that changed it pushed the prediction too high. The result depends on what the decline rule hid, not on choosing a more elaborate model.
The decline rule hid the field's risk pattern
The card scorecard could not learn how missed payments related to bad outcomes from approved accounts because the decline rule had already removed the accounts with those missed payments. The information value chart compares the field across all accounts and approved accounts.
Across all accounts, payment status last month was the strongest field. Among approved accounts, its information value sat at the usual drop line. That is selection, not evidence that the field has no relationship with risk.
Among the approved accounts, the field mostly showed no use of the card, payment in full or revolving. Their bad rates did not form a clear trend. Accounts behind on payments, which carried the risk pattern, were in the declined group. The scorecard's approved-only view had no examples from those values to learn from. A plain logistic regression even read further behind as safer on the approved data, the reverse of what the full outcomes showed.
That is why the inference methods miss on the card data. They begin with a scorecard trained on a population where the decline rule has erased the relationship the score needs to estimate.
A random test arm measures the declined group
Approved-only data cannot resolve the selection problem. To measure the declined group, approve a random share of the applicants each decline rule would otherwise reject, label those loans as the test arm, then wait for their outcomes. Do this separately for each rule because each one selects a different group.
The table shows how precisely a test arm measured the card group's bad rate at different sizes.
| Share approved | Loans | Measured range |
|---|---|---|
| 2% | about 80 | 41% to 60% |
| 5% | about 200 | 45% to 56% |
| 10% | about 400 | 46% to 54% |
Across repeated draws of arms from the declined card accounts, an arm of about two hundred loans measured the group's bad rate within a range of about eleven percentage points in most draws. A smaller arm left a wider range, and a larger arm narrowed it. The useful size depends on the group you need to measure. Use the champion challenger calculator to size the test for your decision. Consider how long the test needs to run with the guide to champion challenger decision strategies.
Keep the arm separate for measurement. In our test, adding a small arm only to train the scorecard did not make the WoE scorecard's prediction match the real rate. A model can use arm outcomes in its build and still fail to measure the group's bad rate. The arm's observed outcomes are the evidence.
Compare the prediction with the arm's measured range
Once the arm has outcomes, compare its measured bad rate with the scorecard's prediction for the same declined group. The figure puts each prediction against the range measured by a random test arm.
On the card data, the scorecard's predicted rate fell well below the arm's measured range. On HELOC, its prediction sat inside the range. The first scorecard could not price the card group, while the approved-only scorecard was already close for the HELOC group. This is the comparison that distinguishes the two cases.
Recalibration fixes the average, not the ranking
When the scorecard's predicted level misses the arm's measured rate, you can recalibrate the score for that declined group. Score the arm's loans, then shift the score's level so its average prediction matches the arm's measured bad rate. That changes the level, not the ranking of applicants within the group.
The table shows the scorecard's prediction before and after recalibration, alongside the real bad rate in our experiment.
| Dataset | Before | After | Real |
|---|---|---|---|
| Card data, about 210 loans | 18.9% | 50.6% | 50.4% |
| HELOC data, about 66 loans | 71.6% | 70.7% | 72.4% |
In both datasets, recalibration brought the average prediction close to the real rate. But matching the group's average does not show whether the score can tell better risks from worse risks inside that group. A corrected average alone is not a reason to let the score replace the decline rule.
Use Gini on the arm to decide whether the rule can give way
Read the score's Gini on the test arm to see whether it ranks risks inside that declined group. The table shows the range of Gini values across random arms.
| Dataset | Arm | Gini range |
|---|---|---|
| Card data | 5%, about 200 loans | -0.16 to 0.11 |
| Card data | 10%, about 400 loans | -0.12 to 0.06 |
| HELOC data | 5%, about 66 loans | 0.18 to 0.65 |
| HELOC data | 10%, about 130 loans | 0.26 to 0.58 |
On the card data, the score's Gini was near zero even with a larger arm. It could not rank good and bad risks inside that declined group, so recalibration did not make the score a substitute for the rule. On HELOC, the Gini was above zero, showing the score could rank the declined group. With the level recalibrated, a cut-off could replace the rule.
Before changing the HELOC rule, backtest the proposed release against past production decisions and inspect the applications it would newly approve or decline. Use the swap set calculator when you assess which applications would switch sides.
How to do reject inference on your book
At the card-data variable screen, do not keep or drop a field on its approved-only information value alone when the decline rule has selected away the values that reveal its relationship with risk.
Use this procedure for each decline rule:
- List the decline rules you need to assess.
- For each rule, approve a small random test arm through a random split in the decision flow. Label the loans and track their outcomes.
- When the outcomes mature, compare the arm's measured bad rate with the score's prediction for that group.
- Recalibrate the score's level.
- Read the score's Gini on the arm. If it is near zero, keep the decline rule because the score cannot rank risk inside the group. If it is clearly above zero, backtest the proposed change and the accounts it would newly approve before letting a cut-off replace the rule.
Where other fields record the same history, start from the approved-only scorecard. Reject inference methods move that estimate according to their assumptions.
Questions about reject inference
Which reject inference method works best?
The methods were not close to the card group's real bad rate in our test. On HELOC, the approved-only scorecard was already close, and the methods that adjusted it moved the prediction too high.
Why does a strong field show a low information value on approved loans?
The decline rule removed the values that carried the risk pattern. Among approved card accounts, the field showed little variation in bad rate, even though it was strongly predictive across all accounts.
Does fuzzy augmentation correct the bias?
Not in our test. It adds declined rows using the first scorecard's predicted probabilities, then feeds those predictions back into a rebuilt scorecard.
How many declined applicants does a test arm need?
Across repeated draws from the declined card accounts, a five-percent arm of about two hundred loans put the measured bad rate between 45% and 56% in 90% of draws. The needed size depends on the group being measured.
Does gradient boosting avoid the problem?
No. In the card data it predicted a 29.4% bad rate for the declined group, against a real rate of 50.4%.
