Skip to main content
Version: 10.2.9

Import data management configurations via Asset Bundle

The article details how to import database and S3 data management configurations as part of an Asset Bundle.

Import database data management configuration​

In an Asset Bundle, the database data management configuration is located in the db-data-purge folder as shown below. For the complete Asset Bundle structure, refer to the guide.

├── db-data-purge
│   ├── config1.json
│   └── config1.sql
└── meta-info.json

A database data management configuration is a JSON file containing the following parameters. The required ones are marked with an asterisk (*):

  • name*: data management configuration name.

  • description: any additional information to describe the configuration.

  • cronExpression: cron expression for scheduled execution.

  • active: if true, the data management configuration is enabled and ready to be started.

  • customSQL: custom SQL script with cleanup logic. The parameter is applicable only for Data Store purging. It is mutually exclusive with the storedProcedureName parameter, meaning you either set customSQL or storedProcedureName with parameters.

    Alternatively, you can save the custom SQL script in a separate file and specify the filename in the customSQL field in the file://config1.sql format. For Data Stores, specify full table names with the schema name. For instance, if a Data Store name is document, the custom SQL should have it like ds.ds_document.

  • storedProcedureName: name of the stored procedure. The parameter is mutually exclusive with the customSQL one, meaning you either set storedProcedureName with parameters or customSQL.

  • parameters: list of parameters for the stored procedure. Populate it only if the data management configuration is based on a stored procedure.

For Asset Bundle import instructions, refer to the following topics:

Important considerations​

To import a database data management configuration successfully, pay attention to the following considerations:

  • If a data management configuration is based on a stored procedure, a Database Analyst should create the procedure in advance.

  • If a data management configuration with the same name already exists, it is overwritten with the new one from the imported Asset Bundle.

  • If the user who imports the Asset Bundle has the Run Data Management Configurations permission, the data management configuration is enabled or disabled as specified in the file imported with the bundle (see the active parameter in JSON). Otherwise, the imported data management configuration is disabled. For details about permissions, refer to the Manage Control Tower roles and permissions topic.

Import diagram​

The following diagram illustrates the import of a database data management configuration:

Configuration file examples​

  • Database data management configuration based on custom SQL with an included script:

    {
      "name": "Test Data Purge SQL Inline",
      "description": "Should clear data store 'test_dp_import_ds2' using inlined script",
      "cronExpression": "* * */1 * * ? *",
      "active": true,
      "customSql": "delete from ds.ds_test_dp_import_ds2;"
    }

  • Database data management configuration based on custom SQL with a script in a separate file:

    {
      "name": "Test Data Purge SQL File",
      "description": "Should clear data store 'test_dp_import_ds1' using script from 'first_db.sql'",
      "cronExpression": "* * */1 * * ? *",
      "active": true,
      "customSql": "file://config1.sql"
    }

  • Database data management configuration based on a stored procedure:

    {
      "name": "Test Data Purge SQL File",
      "description": "Should clear data store 'test_dp_import_ds1' using script from 'first_db.sql'",
      "cronExpression": "* * */1 * * ? *",
      "active": true,
      "storedProcedureName": "dp#custom_purge_sp",
      "parameters": [{name: "i_session_duration_min", value: "60"}]
    }

Import S3 data management configuration​

In an Asset Bundle, the S3 data management configuration is located in the s3-data-purge folder as shown below. For the complete Asset Bundle structure, refer to the guide.

├── s3-data-purge
│   └── config1.json
└── meta-info.json

An S3 data management configuration is a JSON file containing the following parameters. The required ones are marked with an asterisk (*):

  • name*: data management configuration name.

  • description: any additional information to describe the configuration.

  • cronExpression: cron expression for scheduled execution.

  • active: if true, the data management configuration is enabled and ready to be started.

  • bucket*: S3 bucket to clean up.

  • path*: path prefix for the cleaned S3 bucket.

  • fileNamePattern*: Unix-like pattern to filter the files to clean up.

  • recursive*: if true, the configuration cleans up sub-directories.

  • maxFileAge*: parameter specifying that the configuration is to delete files older than its value.

  • sessionDuration*: the maximum purging session duration.

  • accessKeyAlias: S3 access key alias in Secrets Vault.

  • secretKeyAlias: S3 secret key alias in Secrets Vault.

For Asset Bundle import instructions, refer to the following topics:

Important considerations​

To import an S3 data management configuration successfully, pay attention to the following considerations:

  • If accessKeyAlias and secretKeyAlias are not provided for S3, the default system values are used.

  • If there is no entry for accessKeyAlias or secretKeyAlias in Secrets Vault, the data management configuration is disabled.

  • If a data management configuration with the same name already exists, it is overwritten with the new one from the imported Asset Bundle.

  • If the user who imports the Asset Bundle has the Run Data Management Configurations permission, the imported configuration is enabled or disabled as specified in the file imported with the bundle (see the active parameter in JSON). Otherwise, the imported data management configuration is disabled. For details about permissions, refer to the Manage Control Tower roles and permissions topic.

Import diagram​

The following diagram illustrates the import of an S3 data management configuration:

The configuration file example is as follows:

{
  "name": "Test S3 Data Purge",
  "description": "should cleanup test bucket",
  "cronExpression": "* */1 * * * ? *",
  "active": true,
  "bucket": "test",
  "path": "/",
  "fileNamePattern": "*",
  "recursive": true,
  "maxFileAge": "PT1M",
  "sessionDuration": "PT10M",
  "accessKeyAlias": "myKey",
  "secretKeyAlias": "mySecret"
}
troubleshooting

For troubleshooting tips, refer to the following support guides: