The Decisions playbook helps you check whether a decision flow is working and estimate what a change would do before you release it. It uses recorded decisions, outcomes, or a file of records, depending on the analysis. Use the results to decide whether to investigate, test a release, or gather more evidence.
- Choose a decisions analysis
- Check daily movement
- Understand an endpoint's decisions
- Test a release against history
- Compare split traffic
- Review a shadow release
- Find a score cut-off
- Test a flow on records
- Read the result
Choose a decisions analysis
Open New analysis and choose Decisions. The analyses answer different questions:
| Analysis | Use it to | Main answer |
|---|---|---|
| Decision summary | Check daily decision volume and approval movement | Decisions and approval rate for one day compared with the day before and the seven days before |
| Risk report | Understand what an endpoint decided over a period | Approval and bad-rate views across time, segments, amounts, reasons, scores, and outcomes |
| Decision backtest | Estimate what a release would have done to recorded decisions | Decisions that change and how those changed cases performed |
| Champion and challenger | Compare two releases receiving split traffic | Volume, approval rate, outcome rate, and the cases where the arms disagree |
| Shadow review | Gather evidence for a release running beside the live release | Agreement with live, differences, failed requests, and required data |
| Cut-off strategy | Find where a score should be cut under a capacity | The cut-off that pays for itself, including a shared-capacity view by segment |
| Decision flow on records | Test a released flow on history before it has traffic | How its output separates the outcome and what a cut-off on that output pays |
Data sources and record preparation are covered in Data for analysis. Starting an analysis, repeating it, sending findings by email, and opening cases are covered in Running an analysis.
Check daily movement
Decision summary is a morning check. It compares the number of decisions and the approval rate for one day with the day before and the seven days before. You can break the view down by a segment when the overall result needs explanation.
The analysis checks whether approval rate moved beyond a tolerance against the previous seven days. The default tolerance is five points. A warning means the movement deserves investigation. It does not explain the cause by itself.
Choose the decision field, the values that count as approved, and the time field. Use the result to identify a change in daily activity before you examine a release or a segment in more detail.
Understand an endpoint's decisions
Risk report shows what a decision flow decided over a selected period on an endpoint. It reports approval rate by segment and week, and also shows results by amount band, reason, and score band. When outcome data is available, it reports bad rate by month and compares bad rate with approval rate by segment.
Set a floor for approval rate. The analysis warns when approval rate falls below that floor. It reads recorded decisions rather than a sample, so use it to describe the endpoint's decision population for the period.
Compare approval and bad rate by segment to see where decision volume and outcome risk meet. A segment with a different approval rate may need a separate review of its reasons, score band, or outcome history.
Test a release against history
Decision backtest replays recorded decisions through a release. It shows which decisions change and how the cases that changed performed. Use it before releasing a decision flow to understand the effect of the proposed change.
The analysis tests the hypothesis that decisions change. The result gives the number behind the hypothesis and a verdict of held, rejected, or unclear. Treat a small sample as Directional. Some rows may fail in the replay, so separate the changed decisions from rows that could not be replayed when you judge the result.
A backtest is evidence for a release decision. It does not release the flow. After release, deploy a release to an endpoint when you are ready.
Compare split traffic
Champion and challenger compares the recorded decisions from the two arms of a traffic split. It reports volume, approval rate, and outcome rate. You can request the comparison by segment, and you can see the applications that one arm approved while the other declined.
The comparison uses the same population for both arms because the traffic split makes that possible. This makes differences between the releases easier to attribute to the releases themselves. The analysis starts from the endpoint page with its release already selected.
Use the result to decide whether the challenger has evidence to replace the champion. Review the differing applications before making that decision, especially when the aggregate rates are close.
Review a shadow release
A shadow deployment runs a second release beside the live release, and it does not affect the decision. Shadow review compares the shadow with live and reports how often they agreed, how many decisions differed, how many requests the shadow failed, and which data the shadow needed but may not buy.
The analysis tests the hypothesis that the shadow decides as live every time. The result gives a verdict and the number behind it. Use the comparison to gather evidence before the shadow affects anyone.
If the shadow has ended, the numbers are final. When the archive does not keep the shadow's outputs row by row, decisions are compared by count only. Treat that comparison as a limit on what the review can show.
Find a score cut-off
Cut-off strategy looks for where a score should be cut so that what the cut-off stops pays for what it costs, within a capacity. It can find a separate cut-off for each segment.
Provide the saving for each bad record stopped and the cost for each other record stopped. The analysis values every possible cut, finds the best cut overall, and finds the best cut within the capacity you can handle. For segment-specific cuts, the capacity is shared between segments. Each segment therefore cannot assume the full capacity for itself.
The analysis tests whether a cut-off pays and whether one cut-off per segment adds value over one cut-off for all within the same capacity. If the best cut is beyond your capacity, the result also shows the best cut within capacity. Use that constrained cut when the capacity is real.
Few outcomes make the score bands Directional. Treat the economics as uncertain until more outcomes support the comparison.
Test a flow on records
Decision flow on records runs a released decision flow over a file of records. It measures how the output separates the outcome and what a cut-off on that output pays. Use it to test a flow on history before the flow has traffic.
The analysis warns when records failed to score. Check those records before relying on separation or cut-off results, because failed records do not provide a complete view of the flow's behaviour.
Read the result
The result page separates evidence from follow-up:
- Hypotheses states what the analysis tested. Each hypothesis has a verdict of held, rejected, or unclear, with the number behind it.
- Watch-outs records a known trap that the analysis checked. A watch-out can stay quiet, warn, or block the run. Directional means the sample is too small for the number to be firm.
- Findings contains issues that need attention. A finding can be New, Still open, or Cleared.
- Steps is a collapsed detail view of each step the run took and what it returned. It is evidence of how the answer was reached. It is not advice and contains no actions.
- Earlier runs lets you compare the current answer with earlier runs of the same analysis.
Treat a held hypothesis as support for the stated claim, subject to its watch-outs and sample size. Treat a rejected hypothesis as evidence against the claim for the data analysed. Treat an unclear hypothesis as unresolved and look at the supporting number, failed rows, or outcome coverage before acting.