Skip to main content
Version: 4.2

Evaluate historical data

Evelyn has an out-of-the-box solution to validate the model's performance and quality on the client's historical data. A pre-built connector reads the client's report files from a screening system and sends the data to the NSS Business Process (BP) for model evaluation. The report file resulting from the BP execution is stored in S3 Manager for further data and model decision analysis.

Requirements​

Mind the following considerations when working with the historical data evaluation solution:

  • The input file with historical data should be in the CSV structured format.

  • The report columns can have any names.

  • The NSS column mapper file should be completed before the BP is started.

  • A historical data input file row represents full alert information: customer (input) data, hit data, and (optionally) human resolution on the hit.

Workflow details​

Evelyn includes a template format file you should use to map your historical data columns with the NSS standard format. Before running the NSS BP, fill the file with actual column names and update it in S3 Manager. The format file location is doc-upload/name_sanctions_screening/nss_input_format.csv.

Knowing the mapping between the historical data file and the NSS standard format, the NSS BP transforms the original data into a format understandable for NSS and sends it for model evaluation. Your task is to provide a historical data file in the CSV format and upload it to S3 Manager.

Usage instructions​

To use the historical data evaluation feature, follow the steps below:

  1. Download the input format file from S3 Manager: doc-upload/name_sanctions_screening/nss_input_format.csv.

  2. Map the historical data file columns and the standard NSS input format. To do that, update the input format file downloaded in Step 1 with the client column names.

  3. Upload the input format file back to S3 Manager.

  4. Make sure that the original historical data file is in the CSV format and has flat structured data.

  5. Upload the historical data file to any folder of your choice in S3 Manager.

  6. Configure Evelyn to work with the historical data provider. For details, see Install and configure.

    As you set Input data location on the Input step, you can specify one of the following:

    • The path to the file, in which case the BP processes only the specific file.

      Example: If you specify s3://doc-upload/name_sanctions_screening/csv-general-flow-data.csv, Evelyn processes the csv-general-flow-data.csv file.

    • The path to the bucket or folder, in which case the BP takes all existing CSV files under the bucket or folder. The files are processed in parallel, and only one consolidated Quality Check (QC) report is generated.

      Example: If you set s3://doc-upload/name_sanctions_screening/data/, Evelyn processes all CSV files in the data folder and CSV files from the data_october folder located in the data folder.

  7. On the Digital Workers page, in the Name Sanctions Screening group, click the required BP variation and open the CSV File Ingestion BP.

  8. On the Data tab, select No Data.

  9. On the Workflow tab, update the signal ID in the asynchronous NSS-main-BP subprocess if needed.

  10. Go to the Run tab and click Run This Process.

Sample data preparation​

The section details a sample flow to prepare data for the NSS BP execution:

  1. Once the client's historical data file is received for Proof of Concept (PoC) evaluation, upload it to S3 Manager. See a sample historical data file for reference.

    This file contains all required information for the NSS BP execution:

    • Alert ID in column A

    • Customer information (input data) in columns from B to I

    • Hit information in columns from J to Q

    • Manual review decision in column R

  2. Prepare the NSS format file and upload it to S3 Manager. Ensure the following requirements are met:

    • The format file must include all columns that need to be parsed and read from the input file.
    • nss_column should contain only expected Evelyn NSS values. These values must match the Field column entries from the Input record data or Watchlist entity data tables.
    • The client_column values must match the column names used in the input data file. Each line in the format file should correspond to a column in the input file.

    Refer to the sample file for guidance. A typical NSS format file can look like this:

Last updated on