Decision engine demos help you decide whether to buy or build
At the meeting where the demos are booked, an engineer says the credit rules are a few if-statements and that IT could build the engine in a sprint. Next month, vendors will show you their systems, while the engineer expects a response to a much smaller-sounding proposal. You have to decide what to listen for before the first screen share begins.
Test who can change and release policy
Start the decision engine demo with a threshold change. In the Decisimo example, the risk team changes policy, runs tests and impact analysis, then releases the change. A release can include an approval sequence, and deployment to an endpoint waits for that signature.
Ask whether a risk analyst can change and release a threshold without a developer or a ticket. Ask what approval the release requires. You are testing who can make a policy change and how the lender controls its release.
Review changes before they are saved
Ask to see the diff before saving, then compare two releases. The Decisimo leasing example shows added and modified rows, including a change to the deposit value.
A rule returning a decision is one part of the evaluation. The change also needs a review a credit colleague can make. When a threshold change goes through a developer and generated credit code, nobody on the credit side reads the code that was produced. The diff gives the reviewer a view of what changed.
Replay proposed logic against past decisions
Ask any vendor to rerun last quarter’s decisions against the proposed logic and list the decisions that change. The Decisimo champion-and-challenger demo reports results from its impact analyzer.
| Measure | Demo figure |
|---|---|
| Decisions replayed | 1,001 |
| Approve to decline | 30 |
When outcomes were joined, the demo also reported realized bad rates for each outcome cell. That lets the team assess the changed decisions against observed outcomes, rather than treating the replay count as the whole answer.
Explain a decision using its recorded version
Choose a decision from eight months ago. Ask which policy version ran, which inputs it used and which rule applied. A commit log records code changes. For a particular application, the lender needs the version, inputs and rule behind the decision.
For the Decisimo example, each decision records the revisions that ran and the sources of its inputs, including when a value was not found. A stored release snapshot preserves the logic used for an old decision when the policy is edited later.
Check results storage and execution options
Ask where results are stored, how the logic exports and where execution can run. Results in the Decisimo example can be written to a table in the lender’s own Postgres or Supabase, and the logic exports as a Markdown file. Execution can run on regional real-time endpoints or in batch over FTP, S3 and GCS.
If the lender leaves the vendor, its results remain in its own Postgres or Supabase, and it can export its decision logic as a Markdown file.
Compare prices at the same decision volume
Find out which decisions the vendor’s meter counts, then compare the price at today’s volume with the price at three times that volume. Decisimo’s pricing terms count production decisions. Each record in a batch counts, and the number of rules, sources or models used in a decision does not add to the count.
| Measure | Volume | Price |
|---|---|---|
| Standard monthly plan | 50,000 included | From €500 |
| Additional decisions | Each decision above the included volume | €0.01 each |
| Standard plan at 50,000 decisions a month | 50,000 decisions | €500 |
| Standard plan at 150,000 decisions a month | 150,000 decisions | €1,500 |
The price comparison is useful once both quotes use the same definition of a decision. Check the meter as well as the total.
Test the in-house proposal against vendor demos
Building the rule engine is the cheap part. Have the engineering team walk through a threshold change from request to release, show how a credit reviewer reads the diff, and replay the proposed change over past decisions. Then ask them to explain an old decision from its recorded version and inputs. The work continues after the first rule runs: a bureau API can change and a model may need replacing, so ask how each change is reviewed and replayed. The rule evaluator is only one part of the proposal.
Use the same tests in every demo
Use these checks to compare the vendors with the in-house proposal.
- Ask the vendor to have a risk analyst change one threshold and release it, with no developer and no ticket. Ask what approval the release requires.
- Ask to see the diff before it is saved, then compare two releases.
- Ask for last quarter’s decisions to be rerun against the proposed logic and for the decisions that change to be listed.
- Pick a decision from eight months ago and ask which version decided it, on which inputs and through which rule.
- Ask where results are stored, how the logic exports and where it can run.
- Confirm what the price meter counts, then compare today’s volume with three times that volume.
- Ask the in-house team to demonstrate the same change, review, replay and decision-recording work, including what happens when a bureau API changes or a model is replaced.
Questions about decision engine demos
What should you look for in a decision engine demo?
Check who can change and release a threshold, whether a reviewer can read the diff before saving, whether proposed logic can be replayed, whether an old decision can be explained against the version that ran, where results and logic live, and how price changes with volume. Have the vendor demonstrate each check.
Should a lender build a decision engine in-house or buy one?
Building the rule engine is the cheap part. Generated credit code has no reviewer on the credit side, and someone still has to maintain it when a bureau API changes or a model needs replacing. Ask the in-house team and the vendors to demonstrate the same requirements.
Why is git history not enough to explain a credit decision?
Git history records code changes. To explain a particular decision, the lender also needs the policy version, inputs and rule that ran. Ask how the system records those for each decision.
How do you compare decision engine pricing as volume grows?
Ask what the meter counts, then compare the quoted price at current volume with the price at three times that volume. Use the same definition of a decision for both quotes and check how batch records are counted.