Apply continuous learning

Overview

The Continuous Learning Data Store (CLDS), also referred to as continuous learning or decision reapplication, enhances Tara's processing of payment sanctions screening (PSS) alerts by identifying alerts containing the same hits as previously received alerts. If the hits associated with such alerts have been released a configurable number of times over a configurable time interval by a configurable number of different analysts, Tara automatically releases the alerts, bypassing all further processing by downstream Tara models and logic.

The default CLDS settings are as follows:

• CLDS compares incoming alerts with those received over the previously configured number of days.

• CLDS automatically releases any alerts where hits satisfy the following conditions:

  • The hits have never been escalated during the specified time interval and have been released at least the specified number of times by at least the specified number of different analysts.

  • Hit hashing is enabled. If the configuration for minimum releases and anylysts is modified, CLDS is re-evaluated.

• If configured, CLDS automatically escalates any alerts where hits meet all of the following conditions:

  • The hits have never been historically released.

  • Hit hashing is disabled (an operator can manually disable hashing).

The CLDS functionality depends on regular uploads of historical alert decision data. This data is supplied as two CSV files uploaded to a designated bucket in the S3 storage. Currently, CLDS supports PSS alerts sent in the following message formats: FUF (Firco Unique Format), SWIFT_MT, SWIFT_MX, and SWIFT_NPP (the Australian standard New Payments Platform). Tara can process any XML-formatted content with properly supplied input.

To support CLDS, Tara includes an additional Business Process that ingests the data from the message and hit data CSV files and populates two Data Stores: the historical hash Data Store and the hash analytics Data Store. These Data Stores enable CLDS to associate alerts with the hits they contain and maintain a unique hash value per hit. The process includes:

• Computing the hash value using the content of specific fields.

• Collecting the hash counter, analyst names, and the is all released flag, then writing the data to the hash analytics Data Store.

When CLDS is enabled, upon receiving each message containing a PSS alert, Tara invokes the CLDS Business Process step that applies the following logic for each hit in the alert:

1. Computes the hash value using the content of specific fields.

2. Searches the hash analytics Data Store for an entry with a matching hash value.

3. If a matching hash value is found, determines whether the hit was previously associated with an alert that was always released within the configured time interval (365 days by default), at least the minimum number of times (five times by default), and by at least the minimum number of different analysts (two analysts by default).

4. After identifying matching hash values and corresponding alerts, compares message-level tags to filter out messages that are not identical. The tags used for the message-level comparison and other CLDS configurations must be set before historical upload. All configured fetched values are used in computing the hit hash (the default is the TYPE tag).

• If enabled, HIT_TAG_CONTENT is fetched from the message content using a tag name.

• SUBTYPE is fetched directly from the input.

• Resolved decision types are set.

5. Upon determining that the alert contains any hit that does not satisfy the criteria, passes the message to the next Business Process step. If the alert contains only hits that satisfy the criteria, it is automatically released, bypassing further Business Process steps. All hits are evaluated independently.

Mind the following special cases:

• Alert-level CLDS. All hits in an incoming message must meet the criteria for reapplication. Otherwise, all hits proceed through normal Tara processing.

• Hit-level CLDS. Only hits that are unsuitable for reapplication proceed through normal Tara processing.

Add historical decisions

The execution flow looks as follows:

Run CLDS Business Process

To use the CLDS feature, add the decisions to the Control Tower Data Stores. For this purpose, the Ingest Historical Decisions For Continuous Learning Business Process (BP) is included in Tara's bundle.

In the BP, there are only two steps:

• Fetch historical decisions
• Save historical decisions

tip

To run the BP automatically as frequently as needed, create a schedule. For more details, see Schedule Business Process.

Fetch historical decisions

The step locates historical decisions (CSV files) in S3 and converts them into a standard flat format CSV file appropriate for saving data to relevant Data Stores. Two historical files are expected, representing messages and hit information:

• Message file

• Hit file

The exact implementation of the step varies, depending on how you store historical decisions. Currently, data columns match the standard Fircosoft DB extract.

A resulting CSV file is uploaded to S3, and its location is sent to the next step. To avoid memory issues, you might have more than one CSV file with results, currently allowed 100K records per file.

If an alert in the message file does not have corresponding hits in the hit file, such records are ignored.

Historical file name format

Mind the requirements for naming historical files:

• Names of message files must start with messages followed by a unique identifier such as date, for example, messages-01012025.

• Names of hit files must start with hits followed by a unique identifier such as date, for example, hits-01012025.

• A unique identifier must be the same in both filenames.

• The supported file format is CSV.

note

The message_decision value is set to RESOLVE or NO_DECISION according to the provided configuration.

Below are sample files you can use for testing:

• Message file and hit file to test the historical upload functionality

• Decision reapplication input file to test CLDS real-time processing

Historical file columns

Message file

info

Not all the columns below are used in decision reapplication. Columns marked as Mandatory must be present in a CSV input file.

The expected columns in a message file are as follows:

Column name	Meaning	Description	Example
T_DECISION_TYPE	MESSAGE_ANALYST_DECISION	Mandatory column. Can be empty when the T_IS_RELEASED value is used. Contains the analyst's decision.	STP_Released
T_LASTOPERATOR	MESSAGE_ANALYST_NAME	Mandatory column. Must have a value. Contains the name of the analyst who made the decision.	e625124
T_MESSAGE	MESSAGE_CONTENT	Optional column. May not be available in all screening systems. Contains the original message content. If missing, the T_TAG_CONTENT hit level must be present.
T_COMPLETED	MESSAGE_DECISION_DATE	Mandatory column. Contains the timestamp when the decision was made.	2026/03/28 11:10:43
T_MESSAGE_ID	MESSAGE_ID	Mandatory column. Must have a value. Contains the alert ID.	QU2312283S9543000I0001
T_TYPE	MESSAGE_ORIGINAL_SUBTYPE	Mandatory column. Used in calculating the hash. Can have an empty value.	304
T_NATURE	MESSAGE_ORIGINAL_FORMAT	Optional column. Can be used to set a message type. Can have an empty value.	SWF
T_CONTENT_FORMAT	MESSAGE_CONTENT_FORMAT	Optional column. Contains the expected format of the message content. The column might not exist. Used to get the tag content from the message content. The default value is FUF.	FUF
T_IS_RELEASED	IS_RELEASED	Optional column. Boolean. Specifies if an alert was released by analysts. If not present, T_DECISION_TYPE is used. The column might not exist.	true
T_SYSTEM_ID	SYSTEM_ID	Mandatory column. Contains a link between the message and hit files.
T_COMMENTS	DECISION_COMMENT	Optional column. Contains analyst's comments explaining why the historical decision was released, resolved, rejected, or escalated.	ENT IN DUBAI NOT IRAN meaning the analyst determined that the hit entity was located in the city of Dubai, not in the sanctioned country of Iran

Hit file

The expected columns in a hit file are as follows:

Column name	Meaning	Description	Example
SEQNUMBER	HIT_ID	Mandatory column. Must have a string value. Contains a unique hit ID in the message.	0 or hit-123
SYSTEM_ID	HIT_MESSAGE_ID	Mandatory column. Must have a value. Contains a link between the message and hit files.
TAG	HIT_TAG_NAME	Mandatory column. Contains tag name information. Used in calculating the hash.	SWF_4_79
LIST_MATCHING_STRING	HIT_TEXT	Mandatory column. Contains a matched part of the tag content that caused the hit. Can be empty. Used in calculating the hash.
HIT_TYPE	HIT_TYPE	Mandatory column. Can be empty. Used in calculating the hash.	NAME
CITY	HIT_WLE_CITY	Mandatory column. Might contain an empty value.
COUNTRY	HIT_WLE_COUNTRY	Mandatory column. Might contain an empty value. Used in calculating the hash.
ID_LIST	HIT_WLE_ENTRY_ID	Mandatory column. Might contain an empty value. Contains a unique ID of the sanctioned record. Used in calculating the hash.	WCP7491861
RECORD_MATCHING_STRING	HIT_WLE_NAME	Mandatory column. Might contain an empty value. Contains the sanctioned name that matched and is used to calculate the hash.
NAME	HIT_WLE_PRIMARY_NAME	Mandatory column. Might contain an empty value. Contains the sanctioned primary name and is used to calculate the hash.
STATE	HIT_WLE_STATE	Mandatory column. Might contain an empty value.
TYPE	HIT_WLE_TYPE	Mandatory column. Might contain an empty value.	C
T_COMMENTS	DECISION_COMMENT	Optional column. Contains analyst's comments explaining why the historical decision was released, resolved, rejected, or escalated.	ENT IN DUBAI NOT IRAN meaning the analyst determined that the hit entity was located in the city of Dubai, not in the sanctioned country of Iran
T_IS_RELEASED	IS_RELEASED	Optional column. Boolean. Indicates whether a hit was released by an analyst. If the column is absent, T_DECISION_TYPE is used instead. When present, this column takes precedence over the message-level type.	false
T_DECISION_TYPE	HIT_ANALYST_DECISION	Optional column. Contains the analyst's hit-level decision. When present, this column takes precedence over the message-level type.	STP_Released
T_TAG_CONTENT	TAG_CONTENT	Optional column. Contains the hit tag content. When absent, the tag content is parsed from the T_MESSAGE original message content using the TAG and MESSAGE_CONTENT_FORMAT columns.

HIT_WLE_TYPE value conversion

See the HIT_WLE_TYPE value conversion:

("O", EntityType.LOCATION);
("LOC", EntityType.LOCATION);
("LOCATION", EntityType.LOCATION);
("C", EntityType.ORGANIZATION);
("COMPANY", EntityType.ORGANIZATION);
("ORG", EntityType.ORGANIZATION);
("ORGANIZATION", EntityType.ORGANIZATION);
("I", EntityType.INDIVIDUAL);
("IND", EntityType.INDIVIDUAL);
("INDIVIDUAL", EntityType.INDIVIDUAL);
("PERSON", EntityType.INDIVIDUAL);
("V", EntityType.VESSEL);
("VESSEL", EntityType.VESSEL);

HIT_TYPE values

Expected HIT_TYPE values are as follows:

• UNKNOWN
• NAME
• COD
• PASSPORT
• BIC
• EMBARGO
• NATIONAL ID
• FML
• EMC

MESSAGE_CONTENT_FORMAT values

Expected MESSAGE_CONTENT_FORMAT values are as follows:

• FUF (default)
• SWIFT_MT
• SWIFT_MX
• SWIFT_NPP

Converted CSV file columns

The converted CSV file columns are as follows:

• id
• type
• hit_id
• content
• contentformat
• gold_decision
• gold_decision_comment
• gold_decision_date
• analyst_name
• hit_tagname
• hit_hittext
• hit_hittype
• type
• subtype
• hit_sanctiondata_name
• hit_sanctiondata_type
• hit_sanctiondata_primaryname
• hit_sanctiondata_country
• hit_sanctiondata_city
• hit_sanctiondata_state
• hit_sanctiondata_sourceentryid

Extracting data from FIRCO v5 historical tables

select msg.T_SYSTEM_ID,
msg.T_MESSAGE_ID,
msg.T_TYPE,
msg.T_MESSAGE,
msg.T_NATURE,
'FUF' as T_CONTENT_FORMAT,
NULL as T_IS_RELEASED,
actions.T_DECISION_TYPE,
actions.T_COMMENTS,
actions.T_DECISION_DATE,
actions.T_OPERATOR_DESC,
msg.T_LASTOPERATOR,
msg.T_COMPLETED
from FircoContinuity.dbo.FOFA_HIST_MESSAGE as msg
INNER JOIN FircoContinuity.dbo.FOFA_HIST_ACTION as actions
ON msg.T_SYSTEM_ID = actions.T_SYSTEM_ID
WHERE actions.T_DECISION_TYPE NOT IN ('FILTER', 'NO HIT')
AND msg.T_COMPLETED >= Dateadd(Month, -6, GETDATE())
ORDER BY 1;
 

select
hits.SEQNUMBER,
hits.SYSTEM_ID,
hits.TAG,
hits.LIST_MATCHING_STRING,
hits.HIT_TYPE,
hits.CITY,
hits.COUNTRY,
hits.ID_LIST,
hits.RECORD_MATCHING_STRING,
hits.NAME,
hits.STATE,
hits.TYPE
from [FircoContinuity].[dbo].[FOFA_HIST_MESSAGE] as msg
INNER JOIN [FircoContinuity].[dbo].[FOF_HIST_HITS] as hits
ON msg.T_SYSTEM_ID = hits.SYSTEM_ID
WHERE msg.T_COMPLETED >= Dateadd(Month, -6, GETDATE())
ORDER BY 2, 1

Save historical decisions

The step batch-loads decisions to relevant Data Stores using a URL to the Tara-formatted CSV file (currently 1000 hits per batch). All sensitive data in the Data Stores is hashed and does not contain any readable information. All historical data is saved in the pss_reapplication_hashcodes_v4 and pss_hash_analytics_v4 Data Stores linked by a hash ID.

During historical records upload, ensure that the same historical records are not saved multiple times and are not part of the analytics. The same data file can be uploaded multiple times, but it will be reconsidered only once.

There is no concept of consecutive releases. Historical data can be loaded in any order, for example, June data, then January, and so on.

The following hit-level information is hashed:

• TAG_NAME
• HIT_TEXT
• HIT_TYPE
• SANCTION_DATA_NAME
• SANCTION_DATA_PRIMARY_NAME
• SANCTION_DATA_COUNTRY
• SANCTION_DATA_ENTRY_ID
• Tag content (optional)

The following configurable message-level data is hashed optionally:

• message Tags
• subtype

Record data analysis

Each time you upload historical data, new records are added to the pss_dr_file_analytics_data_v4 Data Store.

The following data is calculated for each hit during historical uploads:

• hash_count
• analysts
• is_all_released

If the record completion time is greater than the configurable threshold, it is skipped. hash_count and analysts are updated only for released records.

Upload hash status change

You can update the hash status after a manual review. This action overrides the CLDS decision on specific hits associated with the affected hash ID. As a result, you do not need to modify the Data Store directly, avoiding related security concerns. The process is similar to loading historical decisions.

The hash status change file is processed to update the is_hash_enabled, is_auto_disable, hash_enabled_date, and hash_operator columns in the pss_clds_hash_analytics_v4 Data Store.

The file upload requirements are as follows:

• The filename must start with hash_status_change, followed by a unique identifier such as date, for example, hash_status_change-01012025.csv.

• The file must be in the CSV format.

• Upload the file to the same directory as historical files. It is picked up and processed with historical files on the same schedule.

• Upload only one CSV file to S3. If multiple files are found, they are processed in chronological order (earliest first).

All file columns are mandatory. Records with missing columns or empty values will be ignored. The expected columns are as follows:

Column name	Description
hash_id	Hash ID of the existing record from the pss_clds_hash_analytics_v4 Data Store that requires a hash status change
is_hash_enabled	Hash status flag: true or false
operator	Name of the analyst or operator who requested the change

Apply continuous learning

If the alert content format is not FUF, SWIFT_MT, SWIFT_MX, and SWIFT_NPP, the bot step outputs a "decision_reapplied" signal with the FALSE value, and the BP continues to run through all the steps. If a decision is reapplied ("decision_reapplied signal" == TRUE), the workflow execution skips all ML and data manipulation steps and goes directly to the Generate report bot step. Here, it records the decision reapplication and all relevant data and sends out a response as a decision object fully populated with alerts and hits. For more details, see Overview | Core Business Process.

There are two levels of decision reapplication:

• Alert level. All hits of an incoming alert have to be resolved by CLDS. Otherwise, all incoming hits are processed by Tara.

• Hit level. Only hits of an incoming alert that are not resolved by CLDS are processed by Tara. Hits resolved by CLDS go through standard Tara processing.

Identify candidate for continuous learning

Tara fetches all the decisions that analysts make on alerts and hits that match the incoming request. The initial match is based on hashed unique fields and the number of hits in the alert.

The following fields are hashed and must match a historical alert from the pss_clds_hash_analytics_v4 Data Store:

• TAG_NAME
• HIT_TEXT
• HIT_TYPE
• SANCTION_DATA_NAME
• SANCTION_DATA_PRIMARY_NAME
• SANCTION_DATA_COUNTRY
• SANCTION_DATA_ENTRY_ID

Optional configuration fields are as follows:

• Tag content
• Subtype
• List of message tags

For example, you have the following hit:

{
  "hit_tagcontent": "bank of atlantida",
  "hit_tagname": "ORG_NAME",
  "hit_hittext": "atlantida"
  "hit_hittype": "NAME"
  "hit_sanctiondata_name": "Atlantida INC",
  "hit_sanctiondata_primaryname": " Atlantida Solutions INC",
  "hit_sanctiondata_country": "VUT",
  "hit_sanctiondata_sourceentryid": "12546",
}

Tara normalizes values and concatenates them to generate the hash_id hit. Then, Tara queries the pss_clds_hash_analytics_v4 Data Store for a matching hash ID. The result of this query returns one row of information collected during historical data uploads.

Examining returned data, Tara makes a decision whether or not an incoming request is suitable for reapplication.

• Some data in the pss_clds_hash_analytics_v4 Data Store are written or updated in real time, such as:

  • Enabling a hash status when a hash becomes suitable for reapplication

  • Recording the date and configuration state when a hash is enabled

• If the hash is disabled, hits go through regular Tara processing. Only an analyst can disable the hash ID.

• You can optionally configure decision reapplication to automatically escalate hits. Auto-escalation occurs for a hit if the following conditions are met:

  • The hash_counter field in the pss_clds_hash_analytics_v4 Data Store is zero, and the is_all_released is false.

  • The is_hash_enabled field is disabled (or set to false internally by an operator).

caution

The real-time adjudication is configuration-driven. During a historical upload run, settings must match those during data processing.

The CLDS real-time processing looks as follows:

View CLDS Data Stores

The CLDS Data Stores are as follows:

• Hit-level reapplication Data Store
• Hit-level hash statistics Data Store
• Historical upload execution Data Store

Hit-level reapplication Data Store

All historical alerts are written in the pss_reapplication_hashcodes_v4 Data Store. Each row represents a hit.

The information is stored in the following columns:

• id: unique record identifier
• reapplication_data_id: unique alert identifier
• hit_hash_value: hashed sum of parameters
• msg_id: historical alert ID
• hit_id: historical hit ID
• decision_date: date when the original analyst's historical decision was made

Hit-level hash statistics Data Store

Statistics for each hash are calculated during historical uploads and saved into the hash_id, hash_counter, analyst_names, and is_all_released columns of the pss_clds_hash_analytics_v4 Data Store. Other columns are filled during real-time processing when the hash is enabled.

The information is stored in the following columns:

• hash_id: unique calculated hash of the hit. The hash element depends on the CLDS configuration (hit hash, message tags, and tag content).
• hash_counter: number of times each hit hash occurred in the history table.
• is_hash_enabled: hash status. Initially, blank. When continuous learning rules are satisfied, the field is set to true. When continuous learning is no longer satisfied, the field reverts to blank. The value can also be set to false manually or through the hash status update process.
• is_auto_disabled: not used.
• hash_enabled_date: timestamp indicating when the decision reapplication was first applied to the hash by CLDS. The field relates to is_hash_enabled and follows the same update logic. Initially, blank.
• is_all_released: indicates if all decisions for the hash are marked as released and match the configuration, for example, STP_Releases.
• analyst_names: unique names of the analysts who made decisions for the hash ID. Required for rules.
• minimum_alert_counter: number of alerts required to be cleared by continuous learning at the time of enabling (based on the configuration).
• minimum_analyst_counter: number of analysts required to be cleared by continuous learning at the time of enabling (based on the configuration).
• hash_operator: analyst name who requested a direct hash status update.

Historical upload execution Data Store

Records of every historical data upload attempt are stored in the pss_dr_file_analytics_data_v4 Data Store.

The information is stored in the following columns:

• load_date: timestamp of the upload.
• status: execution status (SUCCESS or FAILED).
• number_alerts_total: total number of alert-level records from the message CSV file.
• number_alerts_loaded: number of successfully loaded alert-level records from the message CSV file.
• number_hits_total: total number of hit-level records from the hit CSV file.
• number_hits_loaded: number of successfully loaded hit-level records from the hit CSV file.
• filename: name of the processed file. If the upload fails, the field includes the reason for failure.

tip

For details on CLDS reports, refer to Run screening | CLDS reports.