Analytics Recipes

Analytics recipes are reusable configurations for analyzing batch test results and comparing decision flow releases. Each recipe defines the metrics to measure, how to aggregate them, and, optionally, how to group the data by dimensions.

What is an Analytics Recipe?

An analytics recipe is a saved configuration that specifies:

  • Measures (Slices) - The metrics you want to analyze from your decision output
  • Dimensions - Optional grouping criteria for breaking down results
  • Aggregations - How to summarize the data (sum, average, count, min, max, quartile, decile, ROC, AUC, Gini, distribution shift)

After you create a recipe, you can reuse it across analyses, and you do not need to configure the metrics again.

When to Use Analytics Recipes

Analytics recipes are useful when you:

  1. Run batch tests regularly - Reuse the same analysis configuration to save time
  2. Compare release revisions - See how changes in decision logic affect outcomes
  3. Monitor key metrics - Track the same KPIs across test runs
  4. Perform impact analysis - Measure how rule changes affect specific dimensions

Creating an Analytics Recipe

From the Recipes Management Page

  1. Navigate to /analysis-recipes in your browser
  2. Click "New analysis recipe"
  3. Enter the recipe details:
    • Title - A descriptive name for the recipe
    • Name - A technical identifier (letters, numbers, underscores, hyphens only)
  4. Add one or more measures (slices)
  5. Save with Ctrl+S or the save button

analytics-recipes-create-form.png

Recipe Structure

Each recipe contains one or more slices (measures). A slice defines:

FieldDescriptionExample
NameLabel for the metric, shown in the chart title. It is saved in lowercase with hyphens"Total Amount" is saved as total-amount
MetricJSONPath to the value in the output$.result.amount
DimensionOptional JSONPath for grouping$.result.category
AggregationHow to summarize valuessum, average, count, etc.
Split byOptional second JSONPath. The slice is calculated for each value at this path. Charts show one line per value, with the seven largest values shown separately and the rest grouped as "Other". Tables show one column per value. Available with every aggregation.Payment channel by week
Only where / is one ofOptional filter. The slice counts only rows where the value at the selected path matches one of the listed values. Values are compared as text.Loan type is one of "POS"
BandsOptional numeric cutoffs for grouping a dimension into ranges. Each upper edge belongs to its band, and rows with non-numeric dimension values are left out. Charts and tables keep the bands in order.3000, 5000, 8000
Show asChoose a display for the slice. Stacked shares shows 100% bars for each split value when the aggregation is share. Bubbles uses the slice for vertical position. Choose Across for horizontal position and Sized by for bubble size. Each dimension value is one bubble.Stacked payment-channel shares by week

Other data available to slices

In addition to decision flow output, slices can use:

  • $.__day, $.__week, and $.__month - the day, week, or month when the decision was made.
  • $.__outcome_day, $.__outcome_week, and $.__outcome_month - the day, week, or month the outcome relates to. For a payment default, this is the instalment due date. Otherwise, it is the start of the outcome window or the date a reported outcome was observed.
  • $.outcome.<name> - 1 or 0 for whether a published outcome with that name occurred. $.outcome_value.<name> gives the value of the outcome, such as a payment channel recorded as a reported outcome.
  • $.review.minutes - the minutes the latest review of the call took, from when the case was opened on the call to the review's decision. $.review.decision gives the decision made in that review.

An outcome is counted only after its observation window closes. Recent weeks therefore include only loans whose outcome window has closed.

Aggregation Types

AggregationDescription
sumTotal of all values
averageMean of all values
countNumber of occurrences
minMinimum value
maxMaximum value
quartileDistribution split into 4 groups (25th, 50th, 75th percentiles)
decileDistribution split into 10 groups (10th, 20th, ... 90th percentiles)
ROCROC curve of a score against an outcome. The metric is the score, and the dimension is the actual outcome, 1 or 0
AUCArea under that ROC curve, from 0 to 1
Gini2 × AUC − 1, from −1 to 1
Distribution shiftHow the same rows move between two versions. A number is split into bands at the baseline's deciles, and the chart shows how many rows moved from each band to each other band. A category shows as a table of old value against new value. Used where two versions are compared

Using Analytics Recipes

In Release Analytics

To analyze batch test results for a release:

  1. Click "Analyze" above the results to open "Analyse results"
  2. Click "From recipe" to expand the recipe selection
  3. Choose your saved recipe from the dropdown
  4. The form fills in the recipe's measures
  5. Click "Analyze" to generate charts

Tip: Click the ⚙️ settings icon next to the recipe dropdown to manage recipes in a new tab.

In runs on the execution node

The "Upload and run on node", "From storage" and "Decision archive (production data)" sources on the release's "Batch test" tab each take an "Analytics recipe". For runs on the execution node, the recipe is computed on the node and displayed with the result. Reports over recorded decisions in your own Postgres decision archive are computed inside that database and can read up to 5,000,000 rows, compared with the 50,000 rows read by a run on the execution node. See Releases and Backtesting.

In Impact Analyzer

To compare revisions using the Impact Analyzer:

  1. Open the Impact Analyzer sidebar from any decision component
  2. Select a decision flow, dataset, and revisions to compare
  3. Choose an Analytics recipe from the dropdown
  4. Click "Analyze" to run the comparison

The Impact Analyzer runs both revisions against the dataset and uses your recipe's metrics to visualize the differences.

In the Risk Report playbook

Under More settings, choose Only where and is one of to filter every built-in slice, including the decision count at the top. Set Amount bands to see approval rate by amount band when an amount path is provided. Add a segment to see approval rate by week for each segment. With a segment and an outcome, the report also shows outcome rate by month for each segment and a bubble chart comparing outcome rate with approval rate for each segment.

Managing Recipes

Accessing the Management Page

Open the recipes management page in any of these ways:

  1. From Analytics Modal: Click the ⚙️ icon, "Manage recipes", next to the recipe dropdown
  2. From Impact Analyzer: Click the ⚙️ icon next to the recipe dropdown
  3. Direct URL: Navigate to /analysis-recipes

Editing a Recipe

  1. Go to the recipes list at /analysis-recipes
  2. Click a recipe title to open the edit form
  3. Change the title, name, or measures
  4. Save with Ctrl+S or the save button

Deleting a Recipe

  1. Go to the recipes list at /analysis-recipes
  2. Click the three-dot menu (⋮) next to the recipe
  3. Select "Delete"
  4. Confirm the deletion

analytics-recipes-list.png

Best Practices

  1. Use descriptive names - Choose a name that makes the recipe's analysis clear (e.g., "Loan Approval Metrics by Region")

  2. Start simple - Begin with 2-3 key metrics and add more as needed

  3. Match JSONPaths to your output - Check that the metric and dimension paths exist in your decision flow output

  4. Use dimensions for segmentation - Group by category, region, or status to identify patterns

  5. Choose appropriate aggregations:

    • Use sum for totals (amounts, counts)
    • Use average for rates and percentages
    • Use decile/quartile for distribution analysis
    • Use ROC, AUC or Gini to measure how well a score separates good and bad outcomes, with the outcome as the dimension
    • Use distribution shift to see which rows a change moves, when comparing two versions

Example Recipe

Recipe: Loan Decision Analysis

NameMetric PathDimensionAggregation
total-approved-amount$.result.approved_amountsum
approval-rate$.result.is_approvedaverage
amount-by-risk-level$.result.approved_amount$.result.risk_levelsum
score-distribution$.result.credit_scoredecile

This recipe would generate:

  • A chart showing total approved loan amounts
  • The overall approval rate as a percentage
  • A breakdown of approved amounts by risk level (Low, Medium, High)
  • A distribution chart showing how credit scores are spread across deciles