An analysis is only as useful as the outcome, records and comparisons it receives. This page explains how to choose attributes, segments, time windows and thresholds so the answer reflects a decision you can act on. Choose the inputs in the start form, and put less common choices under More settings.
- Start with the question
- Choose inputs that support a decision
- Compare segments fairly
- Build a scorecard from usable characteristics
- Set thresholds you would act on
- Set honest cut-off costs
- Set stress and appetite
- Choose identifiers and networks
- Check the result before acting
- Checklist before starting
Start with the question
The start form asks for a small number of kinds of input. Choose each one from the decision you need to make:
- Define the outcome you would regret.
- Set the exposure that gives amounts the same basis.
- Date records from the real start of exposure.
- Choose actionable segments and anything to hold constant.
- Supply a score, predicted probability or characteristics for a scorecard when the analysis needs one.
- Set bands, windows and thresholds to match the decision.
- Supply honest costs, gains, capacity, stress and appetite when the analysis uses them.
The data must also pass its mapping check before you trust the result. See Data for an analysis for data intake and mapping. See Running an analysis for starting an analysis and managing its run.
Choose inputs that support a decision
| Input | How to choose it | Why it matters |
|---|---|---|
| Outcome | Choose the event you would regret, such as default, fraud or chargeback. Define which values mean bad. Make the outcome observable for every record on the same footing. Use an outcome that has had time to happen. | A default that takes 90 days to appear cannot be measured fairly on loans from last month. A very rare outcome makes every band directional. The analysis warns when the number of bad cases falls below its floor. Score analyses need a floor on the order of a hundred bad cases. Simple rate analyses can use fewer, and their warning names the floor. |
| Exposure | Choose the amount at risk for each record, such as balance, limit or loan amount. Use the same basis for every comparison. | Rates measured on one basis and prices measured on another produce wrong totals. |
| Start date | Date the record from when exposure begins, such as disbursement. | Dating from the first payment skips the slow start and overstates what accounts collect. |
| Clock | Compare complete periods. Leave out the last half-finished period. | A partial period often looks weak because it has had less time to produce events. |
| Segment | Group records by something you can act on, such as product, channel, region, store, agent, vintage or term. | A segment should identify an action. Do not use a collection status or a flag set after the fact, because it disguises the outcome. Restricted characteristics are never offered. |
| Score | Provide a number where higher or lower risk can be tested. Let the analysis find which direction is riskier. | A score can separate risk in either direction. Never use a score built with the outcome. The result can look excellent yet mean nothing, and the portal warns when separation is too good. |
| Predicted probability | Provide a score that claims to be a probability. | A predicted probability can be checked for calibration as well as separation. |
| Scorecard characteristics | Choose attributes known at decision time, with a plausible link to the outcome and availability for nearly every record. Leave out identifiers, information recorded after the decision and restricted characteristics. | A moderate number of distinct characteristics makes a scorecard easier to explain and monitor. The analysis reports how many it kept and warns when that number is very small. |
| Bands | Use quantile bands for scores. Use custom edges for characteristics with natural cut points. | More bands show finer differences, but each band holds fewer outcomes. The thinnest band must still hold a meaningful number of outcomes. |
| Repeated-identifier window | Set the time within which a repeat means something. Start short and widen it until extra flags stop catching extra outcomes. | A day can suit application velocity. Slow fraud may need a longer window. The right window depends on how quickly the risk can repeat. |
| Stability window | Compare a recent period with a reference window long enough to hold plenty of decisions. | A reference with too few decisions gives an unstable view of change. |
| Threshold | Set each floor or limit where you would genuinely act. | A threshold expresses your appetite. A fired watch-out should mean that you would take action. |
Compare segments fairly
A segment is a way of grouping records. Use the by input for the grouping you want to compare and the within input when another grouping should define the comparison context.
Analyses that rank segments test them together. They allow for the number of segments tested and flag a segment only when it stands out after that adjustment. A segment with fewer than the minimum number of records is not judged. If the result is noisy, raise the minimum first. Too many segments increase the chance that one looks bad by luck.
Use a control when groups differ for a reason that is not the question. Age is a common example. Old accounts have had more time to pay than new ones, so a comparison without age held constant ranks the oldest accounts first.
Performance drivers lets you choose the grouping to compare and the grouping to hold constant. It compares each group with what the other groups of the same kind would be expected to do. A small group is pulled towards the average, and receives credit only as far as its size supports.
Build a scorecard from usable characteristics
Use characteristic strength to screen candidate attributes first. Each candidate receives an information value. A very high information value is a warning, because it can indicate leakage or another problem. It is not a reason to select the characteristic automatically.
Prefer a moderate number of characteristics over many near-duplicates. Keep attributes that were available at decision time, have a plausible relationship with the outcome and occur for nearly every record. Leave out identifiers and anything recorded after the decision. A scorecard with few variables is easier to explain and monitor.
The scorecard analysis reports how many characteristics it kept. It warns when that number is very small. Treat that warning as a reason to review the candidates and the data before relying on the scorecard.
Set thresholds you would act on
Each analysis offers defaults that reflect common practice. Change them to match your appetite and the action the result should trigger. Illustrative defaults include:
- A minimum Gini around 0.3 for a score to count as separating.
- A population stability index above 0.25 as a real shift.
- An approval-rate fall of 5 points day on day as worth a look.
- A minimum AUC around 0.6 for a fitted scorecard to count as separating.
These values are examples of decision limits. Choose a threshold because it changes what you do, rather than because the number appears precise.
Set honest cut-off costs
A cut-off analysis needs two numbers:
- What you save for each bad record you stop.
- What you lose for each good record you stop, or an amount per record.
The best cut-off depends entirely on the ratio between those numbers. Use real figures. Rerun the analysis with a second pair to see how sensitive the answer is.
Add capacity when you cannot stop every record above the ideal cut-off. Capacity can be the largest share you can stop, such as the share a review team can handle. It turns an ideal cut-off into one you can operate.
Set stress and appetite
Expected loss is probability times loss share times exposure. Choose the stress multiple that represents how bad the next year could be. Set the appetite as the loss rate you would not tolerate.
Use the loss share from your own recoveries where you have them. This keeps the expected-loss comparison tied to what happens in your portfolio rather than to an assumed recovery level.
Choose identifiers and networks
For repeated-identifier checks, choose identifiers that should belong to one person or business. Examples include a phone, device, address or tax number.
A value shared by very many records, such as an office address, tells you nothing useful, so the analysis leaves it out and records the choice.
For networks, use ownership links with a weight for control and exposure analyses. Use shared identifiers for rings. The choice depends on whether the question concerns linked ownership or repeated identity.
Check the result before acting
A hypothesis is a statement the analysis tests. Its result has a verdict of held, rejected or unclear, together with the number behind it. Use the verdict and its supporting number to decide whether the evidence answers the question.
A watch-out is a known trap the analysis checks for. It stays quiet, warns or blocks the run. Directional means the sample is too small for the number to be firm. Treat a directional result as a prompt to obtain more records or change the comparison, rather than as a firm ranking.
When an answer looks too good, check for three causes:
- The outcome leaked into the score or characteristic.
- A record was duplicated across files.
- A segment is the outcome in disguise.
The watch-outs are designed to catch these problems. When one fires, fix the data before trusting the answer.
Checklist before starting
Confirm each point:
- The outcome is observed on one footing, and its bad values are clear.
- The start date is the real start of exposure.
- The segments are actionable.
- Enough outcomes exist for the analysis to produce firm numbers.
- The score is independent of the outcome.
- The mapping check passed.