Using Champion x Challenger in decision strategies - Decisimo - Decision Intelligence Services

When starting to use a new decision model for an existing decision strategy, the goal is to verify how the new decision model or strategy is performing better than the previous strategy.

A good practice is to use a Champion x Challenger validation approach, where the goal is to compare the performance of the Champion (i.e., the existing decision model) against the Challenger (i.e., the new decision model).

This approach is especially useful when the new decision model is not yet fully developed, and there are still some uncertainties about its performance.

In high-impact decision strategies such as credit underwriting, Champion x Challenger should be a standard approach. Testing and slow roll-out ensure the stability of the performance of the whole portfolio.

By using a Champion x Challenger validation approach, we can get quick feedback on how the new model is performing, and identify any areas where the model may need to be improved.

How it can be done?

Here is an example of how the Champion x Challenger validation approach can be used in a simple case.

Suppose we have a decision model that decides which offers to give to customers. The model has been in use for some time, and we now want to develop a new decision model that uses a different approach to offer selection.

When we develop the new decision model, we use the baseline historic decision model as the Champion that will continue evaluation against the majority of cases. The new decision model as a Challenger will be used for the evaluation of a holdout.

By the holdout, we mean a sample (usually random) that is evaluated by the Challenger.

If we find that the new model outperforms the existing model, we can then replace the existing model with the new model. The replacement can be a gradual rollout or a big bang. If we find that the new model does not outperform the existing model, we can either improve the new model or continue using the existing model.

Advantage of Champion x Challenger

The key advantage of the Champion x Challenger validation approach is that it allows us to quickly compare the performance of the new model against the existing model or baseline, without having to wait for the new model to be fully developed.

Another advantage is to be able to see the real-life impact of using the model for decision-making as in-vitro simulations are usually much different from real-life decisions.

Practical implementation in a decision engine

Most of the implementations for Champion x Challenger are done using a random number or multiple random numbers. The random number is part of the data object used within the data message used for the execution of a decision flow.

When there is only one part of the decision flow, that is challenged, one random number may be enough.

On the other hand, when there are multiple challenger decision areas, multiple random numbers can and shall be used. Each challenger decision area gets its own random number and is executed with that number. That prevents possible cross-correlations.

At the end of the execution, all decision results, along with the random number that was used, are stored.

Every model within a decision flow can be used for Challenger x Champion. Whether it is new decision table segmentation or just a decision table row. A single rule or whole rule set. Also testing the performance of third-party data against each other can be challenged.

When to especially use Champion x Challenger?

Champion x Challenger should be used especially in cases where the outcome of decisions takes a longer time to unfold.

For example, in consumer lending, a credit risk performance and risk default vintage shape can take half a year to fully show its steepness. In this case, just blindly switching to a new model would not allow seeing whether the vintage performance is impacted by the models or other environmental changes.

By having Champion and Challenger vintage, the evaluation of performance side-by-side is easy and if the sample sizes are statistically significant, there can be less doubt about other impacts.

Another example, where decision outcomes may take longer to unfold is a result of a marketing campaign and customer lifetime value (CLV). In those cases, the decision may be well-designed, but its outlier performance and propensity to take up additional services are not so well-known.

Conclusion

Champion x Challenger is a good validation practice when developing and using new decision models, as it allows us to quickly compare the performance of the new model against the existing model or baseline.

When using Champion x Challenger, we need to make sure that the sample sizes are large enough to ensure statistical significance, and we need to be aware of the potential for selection bias.

If possible, Champion x Challenger should be used especially in cases where the outcome of decisions takes a longer time to unfold.