How many bads a credit model needs
Enter the bads, bad rate and expected Gini. See the degrees of freedom those bads carry, what they buy in each model, and how many bads a planned model needs.
Degrees of freedom
300 bads carry about 28 degrees of freedom at a Gini of 0.50, and every bin, knot and term the build tries spends them, not only the ones that stay in the model.
Planned model needs
The plan spends 80 degrees of freedom against 28; it needs 854 bads, 554 more than on hand, which is 17,080 loans and 20.5 months with the outcome window.
Degrees of freedom by Gini
What the bads buy
| Model | Bads afford | Degrees of freedom spent |
|---|---|---|
| Logistic regression | 28 variables | 1 |
| Binned logistic or scorecard · at 3 bins | 14 variables | 2 |
| Binned logistic or scorecard · at 4 bins | 9 variables | 3 |
| Binned logistic or scorecard · at 5 bins | 7 variables | 4 |
| Binned logistic or scorecard · at 6 bins | 5 variables | 5 |
| Logistic with splines · at 3 knots | 14 variables | 2 |
| Logistic with splines · at 4 knots | 9 variables | 3 |
| Logistic with splines · at 5 knots | 7 variables | 4 |
| Additive MARS | 14 terms | 2 |
| MARS with interactions | 11 terms | 2.5 |
| Gradient boosting or random forest | 2 variables | No count; ten times data |
A variable binned on the outcome spends its bins even when it enters the model as one WoE term. Screening many variables spends the budget before the model is fitted.
Email your result
Run the model where the decisions are made, then test a new one against the current one on past applications before it goes live. Start free
Show the method
C = (Expected Gini + 1) ÷ 2, μ = √2 × Φ⁻¹(C), π = Bad rate Cox-Snell R squared = 1 − exp(−2 × ∫ Σ f(x, y) ln(f(y | x) ÷ f(y)) dx), Score distribution: N(0, 1) / N(μ, 1) Maximum Cox-Snell R squared = 1 − (π^π × (1 − π)^(1 − π))² S₁ = 0.9, S₂ = Cox-Snell R squared ÷ (Cox-Snell R squared + 0.05 × Maximum Cox-Snell R squared) Degrees of freedom = Bads in sample ÷ π × min over S of (S − 1) × ln(1 − Cox-Snell R squared ÷ S) Binned logistic or scorecard, Logistic with splines: Candidate variables × (Bins per variable − 1) Additive MARS: M + 2 × (M − 1) ÷ 2, MARS with interactions: M + 3 × (M − 1) ÷ 2, M = MARS terms Gradient boosting or random forest: Variables for ML = Degrees of freedom ÷ 10 Bad rate precision = ± 1.96 × √((1 − π) ÷ Bads in sample)
The budget is the largest number of parameters that keeps expected shrinkage at 0.9 or more and optimism in Nagelkerke R squared at 0.05 or less, following the minimum sample size criteria of Riley and colleagues. The Gini becomes an R squared assuming the score is normal with the same spread in goods and bads. The factor of ten for gradient boosting and random forests comes from a simulation study by van der Ploeg, Austin and Steyerberg, in which random forests, support vector machines and neural nets needed over ten times the events per variable of logistic regression; gradient boosting was not in that study.
How many bads does your model need
This tool works out how many degrees of freedom your bads carry at the bad rate and Gini you enter. It applies that budget to logistic regression, scorecards, splines, MARS and gradient boosting.
The result shows how many variables, bins, knots or terms your development sample can support. Use it to size a planned model and see how many bads and months of outcomes it needs.
Common questions
How many bads are needed to build a credit scorecard?
The number depends on the scorecard's degrees of freedom, bad rate and expected Gini. This tool works out the bads needed from the larger of its two sample size criteria.
Is 300 bads enough for a credit model?
It depends on the Gini and the kind of model. The same bads carry a different budget at different Gini values, and flexible models need more data than logistic regression.
How many variables can a scorecard have?
The budget depends on the bads and expected Gini. Bins spend degrees of freedom, so the number of variables falls as the number of bins per variable rises.
What do degrees of freedom and events per variable mean?
Degrees of freedom describe the number of parameters a model can use within the sample size budget. Events per variable relates the bads to the variables used, including variables considered before selection.
How much more data does gradient boosting need?
The page uses a rule of ten times the logistic regression requirement for gradient boosting and random forests. That rule comes from a simulation study, and gradient boosting was not included in that study.