Sometimes the question is not what your policy should be, but what it currently is: the decisions remain on record even when nobody documented the policy, an eight-year-old spreadsheet has become the working system, or you are replacing the system that contains it. Recovery reads those records, reconstructs the policy that produced them, and then grades that reconstruction against the same record, decision by decision.
- When to use it
- Starting a recovery
- The answer
- The funnel
- The stages
- Where it disagrees
- Building it as a decision flow
- What recovery cannot do
When to use it
Recovery needs no outcomes; it needs a history of decisions together with the inputs on which they were made, which is precisely what the decision archive holds. Use it to document a policy that has been running without documentation, migrate one from another system, or check whether a flow behaves as its owners believe it does.
If you do have outcomes and want to determine what the policy should be, use Policy mining instead. Both analyses run through the same pipeline, but they differ in what you point them at.
Starting a recovery
Choose Analyses, then Policy mining, then New pipeline. Leave Realized outcome empty, and fill in the following fields:
- Recorded decision path: the location of the decision in the recorded result, usually
$.decision. The hint says it: "Instead of an outcome: recover the logic behind a recorded decision, e.g. $.decision." - Decision that means declined: the exact value that represents a decline, such as
decline.
The rest of the form behaves as it does for any pipeline. Because the target is a recorded decision, every attribute except restricted and identifier ones may carry a rule; a decline for a product reason is part of the actual policy even though it would never be a risk rule.
The answer
The answer begins with "The policy behind 10,000 recorded decisions", followed by a one-line description of its shape: "1 knock-out, 1 eligibility line, a scorecard on 5 characteristics with a cutoff."
It then gives the grade: "Reproduces 99.97% of the record, 99.90% of the 2,965 decisions it never saw." The second number is the important one, as it is with a scorecard, because the recovered policy is graded against rows that were not used to build it. If every decision matches, the page says "Every decision reproduced. The record cannot tell the recovered policy from the one that made it."
When agreement is less than perfect, the page identifies the gap with "3 decisions differ. See where under each stage."
The funnel
Below the answer, a funnel shows where the decisions went and how well each stage performed: the total recorded decisions, the number that were Knocked out, the number that fell Outside eligibility, and the number that were Scored against the cutoff, each with its own agreement measure.
This is the quickest way to read the result. A stage with high volume and low agreement is where the reconstruction is weakest, so it identifies the stage to examine first.
The stages
The recovered policy is written in the order in which it would run.
1. Knock-outs. These are single conditions that decline outright, each expressed as "Decline when bureau score is 560 or under". The evidence for each condition is one click away: "312 declines, all of them declined in the record", or, when the rule is not pure, "312 declines, 96% of the rows it matches were declined in the record". A pure rule is one that the record never contradicts.
2. Eligibility. This stage uses the same form for conditions that belong to product policy rather than risk, such as minimum age, employment type, or a product not offered in a region.
3. Scorecard and cutoff. After the first two stages, the remaining records are scored and the cutoff is stated as a sentence: "Decline at 580 points or under." When a segment warrants its own line, the policy says so: "Decline at 604 points or under for broker." The characteristics appear as points per band with the engine path beside each one, so the score can be read line by line rather than accepted as a number without explanation.
Each stage header includes its own count and agreement, such as "3,374 decisions, 99.00% reproduced".
Where it disagrees
Show the 3 decisions that differ opens the misses, showing the request, the recorded decision, what the recovered policy says, and the stage that produced that result.
Below them, Where the policy still disagrees groups the misses by what they share, showing the number of rows in each group, its share of all disagreements, and its lift. This is the useful part because disagreement is rarely random: a group that concentrates it usually points to a rule the reconstruction could not express, such as a ratio computed from three inputs, a condition about something the archive does not contain, or an override made by a person outside the system.
That list shows what the record cannot tell you.
Building it as a decision flow
Build as a decision flow turns the recovered policy into a real decision flow, with the knock-outs as a rule set, the eligibility lines beside them, the scorecard as a scorecard, and the cutoff as the decision. The decision flow is built and released, and its backtest against the same record is queued automatically, with the result confirmed as "Decision flow built and released. Its backtest against the record is queued."
The backtest is the honest end of the exercise: it runs the built flow over the archived rows and compares its decisions with the recorded ones in the ordinary result format you already know how to read. Links to the flow and the backtest appear beside the button.
You can edit the built flow. Recovery provides a documented starting point that reproduces most of the record; the remaining few percent are a conversation with the people who made those decisions.
What recovery cannot do
- It cannot recover a rule about something the archive never carried, so if a decision depended on a document that a human read, no amount of analysis will find that rule in the available inputs.
- It cannot express arbitrary arithmetic. Ratios are tried as derived quantities and often close most of the gap, but a rule that combines four inputs in a form that nothing offered will appear as a group in the disagreements rather than as a rule.
- It cannot distinguish a policy from its consequences. If the recorded declines were themselves driven by a score that you no longer have, recovery reconstructs the shape of that score rather than the reasoning behind it.
- It says nothing about whether the recovered policy is good. That question requires outcomes, and it is what Policy mining is for.