Skip to main content
Version: 10.2.9

Data Management API

Allows managing of database and S3 data operations.

Authentication​

For Data Management API, form-based authentication is required. For details, read about the WorkFusion REST API security.

Two permissions are essential for working with Data Management API:

  • Run Data Management Configurations: enables using API methods to submit executions and get statuses or events. Thus, with the Run Data Management Configurations permission only, third-party users can already trigger data management configurations, though without access to sensitive system information. The permission is enabled for the Admin and Developer roles by default.

  • Configure Data Management: allows utilizing the entire range of Data Management API methods. The permission is enabled for the Admin role only by default.

To enable the permissions, follow the instructions in the Manage Control Tower roles and permissions guide.

All URLs are to begin with the following pattern: 

https://control.tower.host/api/v1

Database data management​

GET /data-managements​

Returns a list of all available database data management configurations. Each returned configuration is an object comprising the following data:

  • uuid: unique identifier of the data management configuration
  • active: enabled or disabled state
  • creationDate: timestamp to indicate when a specific configuration was created
  • executionId: identifier of the last data management configuration run (if any)
  • customSql: only for the ds schema, the name of the underlying custom SQL script
  • storedProcedureName: for schemas other than ds, the name of the underlying stored procedure
  • parameters: parameters for the underlying stored procedure (if any)
  • submitProcedureDate: timestamp to indicate when the underlying stored procedure (if any) was submitted
  • lastExecutionPoint and nextExecutionPoint: timestamps for the last and next runs of scheduled data management configurations
  • name: data management configuration name
  • description: additional details (if any) to describe the cleanup operation
  • cronExpression: execution schedule (if any)
  • status: data management configuration status

Query parameters​

  • page: requested page number.
  • size: page size.
  • sort: sorting criterion. Available values : NAME, DESCRIPTION, ACTIVE, LAST_EXECUTION_POINT, NEXT_EXECUTION_POINT, CREATION_DATE.
  • sortDirection: sorting order. Available values: ASC (ascending order), DESC (descending order).

Successful response example​

[
  {
    "active": true,
    "creationDate": 0,
    "cronExpression": "string",
    "customSql": "string",
    "description": "string",
    "executionId": 0,
    "lastExecutionPoint": 0,
    "name": "string",
    "nextExecutionPoint": 0,
    "parameters": [
      {
        "name": "string",
        "value": "string"
      }
    ],
    "status": "QUEUED",
    "storedProcedureName": "string",
    "submitProcedureDate": 0,
    "uuid": "string"
  }
]

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/{uuid}​

Returns information about a particular database data management configuration, including:

  • active: enabled or disabled state
  • creationDate: timestamp to indicate when a specific configuration was created
  • submitProcedureDate: timestamp to indicate when the underlying stored procedure (if any) was submitted
  • executionId: identifier of the last data management configuration run (if any)
  • name: data management configuration name
  • description: additional details (if any) to describe the cleanup operation
  • customSql: only for ds schemas, the name of the underlying custom SQL script
  • storedProcedureName: for schemas other than ds, the name of the underlying stored procedure
  • parameters: parameters for the underlying stored procedure (if any)
  • lastExecutionPoint and nextExecutionPoint: timestamps for the last and next runs of scheduled data management configurations
  • cronExpression: execution schedule (if any)
  • status: data management configuration status

Path parameter​

  • uuid: the UUID for the database data management configuration you want to get information about. Required.

Successful response example​

{
  "active": true,
  "creationDate": 0,
  "cronExpression": "string",
  "customSql": "string",
  "description": "string",
  "executionId": 0,
  "lastExecutionPoint": 0,
  "name": "string",
  "nextExecutionPoint": 0,
  "parameters": [
    {
      "name": "string",
      "value": "string"
    }
  ],
  "status": "QUEUED",
  "storedProcedureName": "string",
  "submitProcedureDate": 0,
  "uuid": "string"
}

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/procedures​

Returns a list of available stored procedures to create database data management configurations for schemas other than ds.

No parameters are required for the API method.

Successful response example​

[
  "string"
]

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/procedures/{sp_name}​

Returns the parameters for a specific stored procedure.

Path parameter​

sp_name: name of the stored procedure for which you want to get parameters. Required.

Successful response example​

{
  "parameters": [
    "string"
  ]
}

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/{uuid}/events​

Returns all events for a particular database data management configuration. Each event object includes the following information:

  • eventDate: date when the event occurred
  • level: log level
  • description: event details

Path parameter​

  • uuid: UUID of the requested database data management configuration. Required.

Query parameter​

  • level: log level. Optional.

    Available values: DEBUG, INFO, WARNING, ERROR. The default value is INFO.

Successful response example​

[
  {
    "configurationUuid": "string",
    "description": "string",
    "eventDate": "2021-01-26T09:49:34.235Z",
    "executionId": 0,
    "level": "INFO"
  }
]

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/{data_management_uuid}/events/{execution_id}​

Returns events for a particular execution of a database data management configuration. Each returned event object comprises the following information:

  • eventDate: date when the event occurred
  • level: log level
  • description: event details

Path parameters​

  • data_management_uuid: UUID of the requested database data management configuration. Required.
  • execution_id: ID of the data management configuration run for which event information is requested. Required.

Query parameter​

  • level: log level. Optional. Available values: DEBUG, INFO, WARNING, ERROR. The default value is INFO.

Successful response example​

[
  {
    "configurationUuid": "string",
    "description": "string",
    "eventDate": "2021-01-26T09:20:25.552Z",
    "executionId": 0,
    "level": "INFO"
  }
]

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

GET /data-managements/{uuid}/statuses/{executionId}​

Returns the current statuses for a particular execution of a specific database data management configuration. The following statuses are possible:

  • QUEUED: the execution is in the queue at the execution engine.
  • PROCESSING: the execution is in progress.
  • FAILED: the execution failed on the database side.
  • DONE: the execution was completed successfully.

Path parameters​

  • uuid: UUID of the requested database data management configuration. Required.
  • executionId: ID of the database data management execution for which status information is requested. Required.

Successful response example​

[
  {
    "configUuid": "string",
    "executionId": 0,
    "status": "QUEUED",
    "submitDate": "2021-01-26T09:43:43.271Z"
  }
]

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

POST /data-managements​

Creates a database data management configuration based on a stored procedure or custom SQL as specified in the request body.

Once the configuration is created, it is submitted to a queue under a unique ID (uuid). From the queue, you can launch it using POST /data-managements/{data_management_uuid}/start.

A successful response returns the following information about the configuration:

  • uuid: unique identifier.
  • creationDate: creation timestamp.
  • status: current status. For a new database data management configuration, the status is QUEUED.

For scheduled database data management configurations, a successful response body can also include a timestamp for the next execution (nextExecutionPoint).

Request body​

For the endpoint, the request body is a required DataManagement object to be saved to the Control Tower (CT) database. The object comprises the following parameters:

  • active: defines whether the created configuration is enabled or disabled. Configurations must be enabled to be executed. So, set the value to true.
  • cronExpression: the schedule for running the configuration. With the parameter specified, the configuration is run according to the schedule once you submit POST /data-managements/{data_management_uuid}/start. Set to null to create an unscheduled configuration.
  • storedProcedureName: name of the stored procedure for data cleanup. Used for purging any database schema other than the ds one. For the ds schema, set it to null and specify the customSql parameter instead.
  • customSql: custom SQL code for cleaning up Data Stores. Set it to null and specify the storedProcedureName parameter instead to create a configuration for purging schemas other than the ds one.
  • description: additional details to describe the created data management configuration.
  • name: any random name defined for the configuration.
  • parameters: includes stored procedure parameters if the created configuration is based on one—that means storedProcedureName is specified. Otherwise, set to null.

Request body example​

{
  "active": true,
  "cronExpression": "string",
  "customSql": "string",
  "description": "string",
  "name": "string",
  "parameters": [
    {
      "name": "string",
      "value": "string"
    }
  ],
  "storedProcedureName": "string"
}

Successful response example​

{
  "active": true,
  "creationDate": 0,
  "cronExpression": "string",
  "customSql": "string",
  "description": "string",
  "executionId": 0,
  "lastExecutionPoint": 0,
  "name": "string",
  "nextExecutionPoint": 0,
  "parameters": [
    {
      "name": "string",
      "value": "string"
    }
  ],
  "status": "QUEUED",
  "storedProcedureName": "string",
  "submitProcedureDate": 0,
  "uuid": "string"
}

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

POST /data-managements/{data_management_uuid}/start​

Launches a database cleanup from the specified configuration without a schedule. Once the method is called, the stored procedure or custom SQL underlying the configuration is executed, and an execution ID is returned.

Path parameter​

data_management_uuid: UUID of the database data management configuration to be executed. Required.

Request body​

The request body is a required parameter array. It specifies the names and values of the parameters for the underlying stored procedure if the data management configuration is to be executed for a schema other than the ds one. For ds schema configurations based on custom SQLs, set the array elements to null.

Request body example​

[
  {
    "name": "string",
    "value": "string"
  }
]

Successful response example​

{
  "executionId": 0
}

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

PUT /data-managements/{uuid}​

Updates an existing database data management configuration.

A successful response returns the following information about the configuration:

  • uuid: unique identifier
  • executionId: last execution ID
  • creationDate: creation timestamp
  • status: current status

For scheduled database data management configurations, a successful response body can also include timestamps for the last (lastExecutionPoint) and next (nextExecutionPoint) executions of the configuration.

Path parameter​

  • uuid: UUID of the database data management configuration to be updated. Required.

Request body​

A required DataManagement object to be updated on the CT side, comprising the same elements as described for POST /data-managements.

Request body example​

{
  "active": true,
  "cronExpression": "string",
  "customSql": "string",
  "description": "string",
  "name": "string",
  "parameters": [
    {
      "name": "string",
      "value": "string"
    }
  ],
  "storedProcedureName": "string"
}

Successful response example​

{
  "active": true,
  "creationDate": 0,
  "cronExpression": "string",
  "customSql": "string",
  "description": "string",
  "executionId": 0,
  "lastExecutionPoint": 0,
  "name": "string",
  "nextExecutionPoint": 0,
  "parameters": [
    {
      "name": "string",
      "value": "string"
    }
  ],
  "status": "QUEUED",
  "storedProcedureName": "string",
  "submitProcedureDate": 0,
  "uuid": "string"
}

Failed response example​

{
  "code": "string",
  "errors": [
    {
      "message": "string"
    }
  ],
  "message": "string"
}

DELETE /data-managements/{uuid}​

Deletes an existing database data management configuration.

Path parameter​

  • uuid: the unique identifier of the database data management configuration to be deleted. Required.

S3 (MinIO) data management​

GET /s3-data-managements/{uuid}​

Returns the following information about a specific S3 data management configuration:

  • uuid: unique identifier assigned to the S3 data management configuration.
  • name: name defined for the configuration.
  • description: additional details about the S3 data management configuration.
  • cronExpression: cron expression for scheduled execution.
  • bucket: S3 bucket to clean up.
  • path: path prefix for the S3 bucket to clean up.
  • recursive: if true, the configuration cleans up sub-directories.
  • fileNamePattern: Unix-like pattern to filter the files for subsequent cleaning.
  • maxFileAge: files older than the specified value are to be deleted.
  • lastFileModifiedTime: timestamp when the configuration was last modified.
  • sessionDuration: purging session duration.
  • accessKeyAlias: S3 access key alias in Secrets Vault.
  • secretKeyAlias: S3 secret key alias in Secrets Vault.
  • active: defines whether the created S3 data management configuration is enabled or disabled.
  • lastExecutionPoint or nextExecutionPoint: date when the configuration was executed last or to be executed next, accordingly, if the configuration has a schedule set (cronExpression).
  • creationDate: date when the configuration was created.
  • status: current status of the S3 data management configuration instance.

Path parameters​

  • uuid: UUID of the data management configuration. Required.

Successful response example​

{
  "uuid": "string",
  "name": "string",
  "description": "string",
  "cronExpression": "string",
  "bucket": "string",
  "path": "string",
  "recursive": true,
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": 0,
  "accessKeyAlias": "string",
  "secretKeyAlias": "string",
  "active": true,
  "lastExecutionPoint": 0,
  "nextExecutionPoint": 0,
  "creationDate": "string",
  "status": "string"
}

GET /s3-data-managements/{data_management_uuid}/statuses/{execution_id}​

Returns all statuses for a specific execution of an S3 data management configuration:

  • QUEUED: the execution is waiting in the queue at the execution engine.
  • PROCESSING: the execution is in progress.
  • FAILED: the execution failed on the database side.
  • DONE: the execution was completed successfully.
  • TIMEOUT: the execution timeout for the configuration is exceeded.
  • PARTIAL_SUCCESS: purging removed only some of the files matching the specified criteria.

Path parameters​

  • data_management_uuid: UUID of the S3 data management configuration. Required.
  • execution_id: identifier of a specific configuration execution. Required.

Successful response example​

{
  "configUuid": "string",
  "executionId": 0,
  "startTime": "string",
  "endTime": "string",
  "status": "string"
}

POST /s3-data-managements​

Creates a data management configuration to clean up an S3 bucket (MinIO).

Request body​

A required DataManagement object to be saved to the Control Tower (CT) database, comprising the following parameters:

  • name: any random name defined for the configuration.
  • description: additional details about the created S3 data management configuration.
  • cronExpression: cron expression for scheduled execution.
  • bucket*: S3 bucket to clean up.
  • path*: path prefix for the S3 bucket to clean up.
  • recursive*: if true, the configuration cleans up sub-directories.
  • fileNamePattern*: Unix-like pattern to filter the files for purging.
  • maxFileAge*: files older than the specified date are to be deleted.
  • lastFileModifiedTime: timestamp when the configuration was last modified.
  • sessionDuration*: purging session duration.
  • accessKeyAlias: S3 access key alias in Secrets Vault.
  • secretKeyAlias: S3 secret key alias in Secrets Vault.
  • active: defines whether the created S3 data management configuration is enabled or disabled. Data management configurations must be enabled to be executed. So, set the value to true.

The parameters marked with an asterisk (*) are required.

Request body example​

{
  "name": "string",
  "description": "string",
  "cronExpression": "string",
  "bucket": "string",
  "path": "string",
  "recursive": true,
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": 0,
  "accessKeyAlias": "string",
  "secretKeyAlias": "string",
  "active": true
}

Successful response example​

{
  "uuid": "string",
  "name": "string",
  "description": "string",
  "cronExpression": "string",
  "bucket": "string",
  "path": "string",
  "recursive": true,
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": 0,
  "accessKeyAlias": "string",
  "secretKeyAlias": "string",
  "active": true,
  "lastExecutionPoint": 0,
  "nextExecutionPoint": 0,
  "creationDate": "string",
  "status": "string"
}

POST /s3-data-managements/{data_management_uuid}/start​

Launches S3 data purging based on the specified configuration without using the scheduler and returns its execution ID.

Path parameters​

  • data_management_uuid: UUID of the S3 data management configuration. Required.

Request body​

The request body is a required execution parameter array as shown in the example below.

Request body example​

{
  "path": "string",
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": "string"
}

Successful response example​

{
  "executionId": 0
}

POST /s3-data-managements/{data_management_uuid}/toggle​

Lets you enable or disable an S3 data management configuration.

Path parameters​

  • data_management_uuid: UUID of the S3 data management configuration. Required.

Request body​

When the request body parameter is set to true, the data management configuration specified in the path parameter is enabled. Otherwise, it is disabled.

Request body example​

{
  "active": true
}

PUT /s3-data-managements/{uuid}​

Updates an existing S3 data management configuration.

Path parameter​

  • uuid: UUID of the S3 data management configuration to be updated. Required.

Request body​

A required DataManagement object to be updated on the CT side, comprising the same elements as described for POST /s3-data-managements.

Request body example​

{
  "name": "string",
  "description": "string",
  "cronExpression": "string",
  "bucket": "string",
  "path": "string",
  "recursive": true,
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": 0,
  "accessKeyAlias": "string",
  "secretKeyAlias": "string",
  "active": true
}

Successful response example​

{
  "uuid": "string",
  "name": "string",
  "description": "string",
  "cronExpression": "string",
  "bucket": "string",
  "path": "string",
  "recursive": true,
  "fileNamePattern": "string",
  "maxFileAge": "string",
  "lastFileModifiedTime": 0,
  "sessionDuration": 0,
  "accessKeyAlias": "string",
  "secretKeyAlias": "string",
  "active": true,
  "lastExecutionPoint": 0,
  "nextExecutionPoint": 0,
  "creationDate": "string",
  "status": "string"
}

DELETE /s3-data-managements/{uuid}​

Deletes an existing S3 data management configuration.

Path parameter​

  • uuid: UUID of the S3 data management configuration to be deleted. Required.