Confusion matrix
A confusion matrix is a two-by-two table comparing a fraud or credit check’s results with what turned out true. A true positive is fraud flagged, and a false positive is a good customer flagged. A false negative is missed fraud, and a true negative is a good customer passed.
Example
For illustration, consider 100 payments, with 10 involving fraud and 90 from good customers. At one cutoff, 6 fraud cases and 9 good customers are flagged, while 4 fraud cases are missed and 81 good customers pass. Precision is 6 ÷ 15, or 40%, and catch rate is 6 ÷ 10, or 60%.
Reading the cells and rates
The table crosses what turned out to be true with what the check did: flagged or passed. That places results in the true positive, false positive, false negative or true negative cell.
Precision uses all flagged cases as its denominator. Recall, also called catch rate, uses all fraud cases, and false positive rate uses all good cases. A false positive rate is false positives divided by false positives plus true negatives.
How thresholds change the matrix
At a higher score threshold, fewer cases are flagged. Precision usually rises, and catch rate usually falls.
To choose a cutoff by cost, assign costs to the outcomes. A fraud miss carries fraud loss, and a false decline carries lost margin. Weighting the cells by those costs gives the cheapest threshold.
Questions we get asked
What do the rows and columns of a confusion matrix compare?
They compare what turned out to be true with what the check flagged or passed. The table’s orientation can vary, but the four outcome categories are the same.
Does a confusion matrix explain why one payment was flagged?
No. It groups results by whether the check flagged or passed and what later proved true. It does not give the reason for a particular flag.
Can a confusion matrix tell me what a false decline costs?
No. The matrix shows the outcome category, not its cost. To weigh that outcome, assign a cost such as the lost margin from a false decline.
Decisimo’s fraud threshold calculator finds the fraud score cut-off that saves the most and shows precision, catch rate and a confusion matrix.
