How many bads does a credit model need?
You are about to build a scorecard, or update the one in production. Someone has said you should rebuild it. The book has a few hundred bads. The new model says 5%, and that number will go into the price and the provision.
Before the build, settle two things: whether there is a reason to rebuild and whether you have enough bads for the plan. Quite often, the data cannot carry a new scorecard at all. Three hundred bads is a starting point for a calculation, not a pass mark.
Why the number of bads matters
The 5% your model gives you is not exact. With 300 bads behind it, the underlying bad rate can sit at about 4.5% to 5.5% by design, before the model has aged a day.
That gap goes straight into money. In a demo book with €50m booked in a year and 75% of a bad loan lost, against the 5% the model gives, 4.5% or 5.5% is about €187,500 a year either way. Double the volume and the money in the gap doubles too.
The model can also be wrong in a more damaging way when its build is bigger than its bads allow. It calls safe customers safer and risky customers riskier than they are. The approved book then loses more than planned, and good applicants are turned away. In one demo case, a band the model calls 2% runs at about 3%.
Deterioration after go-live comes on top of this initial uncertainty. The more money and volume ride on a model, the more you need to know how far to trust its bad rates.
What “enough” means
The bads set two limits. They limit how much the model can learn, because every variable and every bin the build tries is estimated from the same bads. Past a point, the fit learns the noise of this one sample.
They also limit how precisely you know the bad rate, both for the whole book and for each band you price on. Statisticians describe the first limit as a budget of degrees of freedom. The expected Gini decides most of its size. The next sections put numbers on both limits.
Is 300 bads enough?
It depends on the expected Gini and on what you plan to build. At a 5% bad rate, 300 bads carry about 9 degrees of freedom at a Gini of 0.3, 28 at 0.5, and 63 at 0.7. The bad rate barely moves that budget. The Gini decides it.
| Expected Gini | Degrees of freedom |
|---|---|
| 0.3 | About 9 |
| 0.5 | About 28 |
| 0.7 | About 63 |
The weaker model needs more bads for each degree of freedom. For 30 degrees of freedom at a 2% bad rate, the requirement is about 130 bads at a Gini of 0.7 and about 930 at 0.3. At a 5% bad rate and Gini of 0.5, 30 degrees need about 320 bads.
The budget comes from the minimum sample size criteria of Riley and colleagues, which keep the model’s expected shrinkage to 10% or less. The model sample size calculator works it out for your own bads, bad rate and Gini.
What spends the budget
Every candidate parameter counts before selection. A variable's coefficient counts. A variable cut into five bins spends four degrees of freedom. A WoE variable counts its bins too because those bins were fitted on the same bads. That is the convention used by the tool. A spline spends its knots minus one.
| Specification | What 28 degrees of freedom buy |
|---|---|
| Continuous variables | 28 |
| Three bins | 14 variables |
| Four bins | 9 variables |
| Five bins | 7 variables |
| Six bins | 5 variables |
| MARS, no interactions | 14 terms |
| MARS, with interactions | 11 terms |
Screening spends the budget too. Forty candidate variables at five bins use 160 degrees of freedom before selection, against the 28 carried by 300 bads at a Gini of 0.5. Variables that do not survive still used capacity during the build.
A planned scorecard against 300 bads
Consider a demo plan with 20 candidate variables at five bins. It spends 80 degrees of freedom. At a 5% bad rate and Gini of 0.5, the plan needs about 854 bads, which is 554 more than the 300 on hand.
| Plan | Degrees of freedom | Bads needed | Months |
|---|---|---|---|
| 20 variables at five bins | 80 | 854 | 20.5 |
At 2,000 loans a month and a 12-month outcome window, those outcomes take 20.5 months to become known. The choice is then practical: reduce the plan, wait for outcomes, or make a smaller change to the current scorecard.
When the bads say no
When the plan outruns the bads, the arithmetic narrows the choices. It does not make the decision for you.
- Keep the current scorecard and recalibrate it. Refitting its intercept and slope spends 2 degrees of freedom and needs about 22 bads at a Gini of 0.5.
- Build a smaller scorecard that stays inside the available budget.
- Wait for the outcomes that the planned model needs.
- Test any new model against the current one on past applications before it replaces it, using the champion-challenger test.
A rebuild needs a reason as well as the bads. “It has been a while” or “we heard we should” is not a reason on its own. The arithmetic says what the bads support. The decision remains yours.
The interval around the bad rate
The number of bads sets how precisely the bad rate is known, whatever the model's strength. At a 5% rate, the 95% intervals are:
| Bads | Interval around 5% |
|---|---|
| 100 | 4.04% to 5.96% |
| 300 | 4.45% to 5.55% |
| 1,000 | 4.70% to 5.30% |
| 3,000 | 4.83% to 5.17% |
A score band has fewer bads than the whole book, so its interval is wider. In a demo band with 30 bads at 12%, the interval is 7.97% to 16.03%. A low-risk band with 15 bads at 1.5% has an interval from 0.75% to 2.25%.
This is the uncertainty on the day the model is built. Deterioration comes later. The operating response belongs in scorecard monitoring.
When the build overspends
Overspending the budget makes predictions too extreme on new data. The expected calibration slope is the shrinkage factor. It pulls each predicted log-odds difference toward the average.
In the demo case at a Gini of 0.3, 300 bads and a 5% average bad rate, spending 80 degrees of freedom gives an expected shrinkage factor of 0.51. A prediction of 2% then runs at about 3.1%.
| Degrees spent | S | 2% runs at | 10% runs at | 20% runs at |
|---|---|---|---|---|
| 9 | 0.90 | 2.19% | 9.4% | 17.7% |
| 28 | 0.75 | 2.52% | 8.4% | 14.5% |
| 80 | 0.51 | 3.13% | 7.2% | 10.5% |
These are expected values under the criteria's assumptions, not observed results. The approved end is riskier than the model says, losses run above plan, and good applicants are declined.
Before you sign off
- Say why you are rebuilding at all.
- Count every candidate parameter the build will try, including bins and screening.
- Compare that count with the budget at the Gini you expect, using the sample size tool.
- Count the bads in each band you will price or cut on.
- Put the interval into the price and the provision, not only the point estimate.
- Validate and test out of time as usual, with the approach described in documenting a credit scoring model.
Validation and out-of-time testing check the model on data it was not fitted on. They do not widen the sample or add bads to the budget.
Frequently asked questions
Is 300 bads enough for a scorecard?
There is no general pass mark. At a 5% bad rate, 300 bads support about 9 degrees of freedom at a Gini of 0.3, 28 at 0.5, and 63 at 0.7. The result depends on the expected Gini and the planned model.
Does a WoE variable count as one degree of freedom?
Its final model representation may show one coefficient, but the bins were fitted from the same bads. The tool therefore counts the bins, using one fewer degree of freedom than the number of bins.
How much more data does gradient boosting need?
The one simulation study found that random forests, support vector machines and neural nets needed over ten times the events per variable of logistic regression; gradient boosting was not in it, so ten times is a floor taken from that study, not a rule.
Does out-of-time validation make up for a small sample?
No. It tests performance on data the model did not fit, which answers a different question. It does not add bads to the development sample or increase the budget of degrees of freedom.
