Data your analyses read

Every analysis needs one of three kinds of evidence: records and their history, recorded decisions and later outcomes, or the whole portfolio at published month ends. This page explains what each kind contains, how the portal prepares it, and how to tell whether it is ready. Your data stays in your own analytics database, and the analyses run there on your node.

Choose the evidence the question needs

The New analysis dialog states the data need for every analysis. It shows the missing piece and a link to fix it, and it puts analyses that are ready first.

If you need to understandThe analysis readsWhy
How accounts pay back over timeRecords, with optional dated events linked to each recordPayments and other events show what happened to each account, application or customer
Whether a decision change helpedThe decision archive and, where required, the outcome historyThe archive shows the decisions taken, while the outcome history shows what happened afterwards
What to provision or how the book is performingPublished portfolio month endsA complete book snapshot shows the state and cash position on one date

Most analyses in Portfolio and losses, Scores and models, and Watching for risk use records. Analyses in Decisions, including Scorecard from decisions and Model stability, use the decision archive and may also require an outcome history, a release, an endpoint, a shadow deployment or a release with a traffic split. See Decision archive and outcome history.

Provision, Provision back-test, Collections, Vintages, Collection queue and Work queue use the portfolio warehouse. The customer's accounts, monthly state and cash enter that warehouse through the Data hub. Each month is published as a month end after its checks pass. Analyses read published month ends only, so a month that fails its checks cannot affect an analysis number.

Provision also uses Coefficients, which contain the loss rates per state in force for a month. Month ends shows which months are published. The Portfolio page is the overview, and Explore lets you slice the book.

For analyses that start from files or a decision archive feed, see Data sources. For starting, repeating, email and case settings, see Run an analysis.

Prepare records

A data intake is a set of your files, or a feed read from the decision archive, that the portal reads, checks and maps into records. Files may be xlsx, csv, JSON lines, or zipped or gzipped versions of those files.

Open Data intake and work through these steps in order:

  1. Files accepts the files.
  2. Check examines whether they fit together and what could change the numbers.
  3. Map assigns meaning to each table and column.
  4. Records writes the mapped records and shows how events matched.
  5. Analyses shows which analyses can use the intake.

Records contain one row per account, application or customer. They may also contain dated events linked to the record. An intake can instead be a feed that reads recorded decisions from the decision archive over a window. A feed refreshes by itself on a schedule.

An intake may also carry networks and observations. Networks describe which records are linked by shared identifiers or ownership, with directed weights. Connection analyses use them. Observations are connections read from a data source, and Connections watch uses them.

Check the files before mapping them

Check reports issues before any analysis runs, so you can fix data problems before they affect a result. It looks for:

  • rows that point to no record
  • the same record repeated across tables
  • columns that use two names
  • values that stand for empty, such as 0 or error text in a date column
  • summary sheets that are not data
  • numbers inside a file name
  • a counter that never changes, which indicates a snapshot rather than a clock

The portal says whether the files fit together. It ranks findings by whether they change the numbers or only need a look.

Dates that can be read in both directions, such as day first or month first, ask you to choose the interpretation.

Map each field's meaning

Map proposes a meaning for each table and column from the files. Confirm the proposal or correct it. A proposal that conflicts with the data is flagged.

A table is one of Records, Events, Lookup or Summary. A column can be:

  • Record key, which identifies the record
  • Start date
  • Exposure, the amount at risk for the record
  • Outcome, with the values that mean the bad outcome
  • Segment, a way to group records
  • a plain field
  • Not used

Events have a date, a value and the key that links them to a record. Records then writes the mapped data and shows how events matched. If more than a few percent of events match no record, analyses warn because the intake usually lacks a file.

Mapping captures the concept behind a column once. Later analyses can then preselect a field by what it means, such as exposure, start, score, predicted probability or outcome. If you skip mapping, the start dialog asks you to identify the required concept for that analysis.

Choose usable segments

Attribute roles determine which fields may be used as segments. Product and channel fields are offered as segments. Restricted characteristics, such as gender or nationality, and identifiers are never offered. See Attribute roles.

A segment is a way of grouping records. Choose a field whose groups answer the question you want to examine and whose values are meaningful in the records. The portal uses the mapping and attribute roles to limit the fields offered for that purpose.

Portfolio data must be published

Portfolio analyses do not read an incomplete month. The portfolio warehouse contains accounts, their monthly state and cash. After the checks pass, a month is published as a month end, and only then can an analysis use it.

If no month is published, an analysis that reads the warehouse says no portfolio data. Use Month ends to see which months are available. Use Explore on the portfolio data pages to examine the book before running an analysis. Provision also needs the applicable Coefficients for the month.

Know what the terms mean

The portal uses outcome for the bad event, such as default, fraud or chargeback. Bad rate is the rate of that outcome. Exposure is the amount at risk for a record. An event is a dated row linked to a record, such as a payment.

If an analysis needs decisions and later results, connect the decision archive and outcome history described in Decision archive and outcome history. If it needs a destination or connection, see Sink connections.