Backtesting

Backtesting means running a strategy or model against historical data to estimate how it would have performed. It helps you assess effectiveness, risk, and consistency before you put the decision logic into production. Always test with the data that was actually available at the time of each decision. Backfilling with today’s data distorts the results and reduces the value of the test.

Example

A lender wants to review a policy that declines applications when monthly debt payments exceed 40% of income. In this illustration, the lender runs that policy over 1,000 past applications, using each applicant’s information as it stood at the time. The results show which applications the policy would have approved or declined.

In champion-challenger testing, the impact analyzer runs the champion and challenger over past decisions or a file and counts which decisions would change.