Champion-challenger tests with a 5% challenger can take 19.6 months
The test
You are comparing a decision model or strategy already in use with a new one before deciding whether to replace the current approach. The champion evaluates most cases in a holdout, while the challenger evaluates a usually random sample. If the challenger performs better, you can replace the champion gradually or at once; if it does not, you improve it or drop it.
That comparison also needs a shared way to identify which approach handled each case. Most implementations carry a random number in the data used by the decision: one number is enough when one part of the decision is challenged, while separate numbers keep several tests from interfering with one another. The results are stored with the number used, so you can identify the two groups when you compare their performance.
You can challenge a decision table’s segmentation, a single row, one rule, a whole rule set, or one third-party data source against another; each is a separate area, so each gets its own number.
The waiting time
The outcome window belongs in the test plan because consumer credit performance can take half a year to show. Keeping the two arms’ vintages side by side lets you distinguish a difference between models from conditions that changed around the same time. Without that comparison, a change in vintages has no clear explanation.
At a 4% bad rate, showing a one-point difference takes 19.6 months with a 5% challenger and a six-month outcome window. The same comparison takes 8.7 months with a 50% challenger and that same window.
| Challenger share | Months to fill | With a six-month outcome window | With a twelve-month outcome window |
|---|---|---|---|
| 5% | 13.6 | 19.6 | 25.6 |
| 10% | 7.2 | 13.2 | 19.2 |
| 20% | 4.1 | 10.1 | 16.1 |
| 50% | 2.7 | 8.7 | 14.7 |
The challenger’s own loan count changes less than the calendar time suggests because the champion arm supplies the comparison. For a one-point difference, the challenger has 3,407 loans at a 5% share and 6,745 at a 50% share; what changes most is how long the smaller share takes to collect them. Seeking a half-point difference extends the wait again: a 5% challenger needs 58.7 months with a six-month outcome window, compared with 16.2 months for a 50% challenger.
The split
A bounded split gives every request a clear destination. With the challenger covering split values 1 to 30 and the champion covering everything else, exactly 30 of 100 requests go to the challenger and 70 to the champion, and every request receives a decision. Add a second challenger covering 21 to 40 and the same boundaries divide 100 requests into 20, 20 and 60 across the three branches. The numbers are simple because the ranges are explicit: each request can be placed in one branch, and the branches do not leave a gap.
That boundary also determines what happens when the request has no split value. The safe pattern bounds the challenger from 1 to 30 and lets the champion catch the rest, so a missing split value goes to the champion. Values outside the expected range are handled there too: a split value of 0 or 101 still receives a decision. The experiment is the thing under test; it is never the thing you fall back to.
The obvious-looking pair of rules reverses that safety. A challenger covering values at most 30 and a champion covering values above 30 appear to cover everything, but a missing value counts as less than every number, so it matches the challenger. For those applications, a 70/30 champion-to-challenger test becomes 0/100. Adding a champion catch-all does not repair the result when the challenger is checked first: with the challenger set to “at most 30” and the champion set to “any value,” the missing value still reaches the challenger. Only bounding the tested branch keeps missing values on the live model.
A split can use a field such as channel. A branch for “broker” and the champion for the rest send broker applications to the challenger and every other channel, including one nobody planned for and a missing one, to the champion. Listing three named channels without a catch-all leaves a fourth value with no decision at all. The same rule applies as in the numeric split: bound the tested branch and give the champion the rest.
The operating check
Before you put the changed table, scorecard or rule set into live traffic, run it against past decisions or a file. The impact analyzer reports how many decisions would change, such as “30 of 1,001 decisions would change,” and lists the old and new outcomes. Once outcomes are joined, you can see the realized bad rate for each group rather than judging the revision from changed decisions alone.
That check does not create the live split for you. On live traffic, a fork in the decision flow uses the split value sent by the lender’s system; there is no built-in random split. The assistant can add the fork with the champion as the catch-all, and each change is approved before it is used. The number that identifies the arm therefore has to arrive in the data your decision receives.
The same number needs to remain attached after the decision. When a decision archive is connected, each decision is written to the lender’s own database with its application document and split value included. You can then compare the two arms against the lender’s own outcomes, with the value used for routing still attached to each decision. That keeps the comparison tied to the cases the lender actually received.
The challenger becomes the champion by deploying its release to the endpoint, and you can compare two releases first.
The next test
Put your expected bad rate, challenger share and outcome window into the champion-challenger calculator. It does the arithmetic using your numbers and checks a finished comparison using the difference, 95% interval and p-value.
Before assigning traffic, check whether the difference you expect is one your planned test length can detect. At a 20% share, a test that takes in loans for 1 month can detect a difference of 2.11 percentage points or more, 3 months 1.18, 6 months 0.82, and 12 months 0.57.
Frequently asked questions
Does a live decision flow create the champion-challenger split automatically?
No. A live decision flow does not create a random split on its own. The lender’s system must send a split value in the data the decision receives, and a fork routes each request using that value. Approve the change before it is used.
How can I check a changed scorecard before sending it live?
Run the changed scorecard against past decisions or a file before sending the revision into live traffic. The impact analyzer reports how many decisions would change and lists the old and new outcomes. The impact analysis does not create a live traffic split.
Can a champion-challenger test compare third-party data sources?
Yes. A champion-challenger test can compare one third-party data source with another. Treat that comparison as a separate test area and assign it its own random number, so its results can be identified separately from other parts of the decision.
How should multiple challenger branches divide requests?
Set explicit ranges for the challenger branches and let the champion cover everything else. That gives each request a destination, including requests with missing or unplanned split values. Listing only the values you expect can leave an application without a decision.
What should a decision archive record for a champion-challenger comparison?
A decision archive should record each decision with the application document and the split value used to route it. The archive connects to the lender’s own database, keeping the arm assignment attached to the cases the lender received so the results can be compared with the lender’s outcomes.
