Configure separate S3 accounts for Control Tower and VDS

caution

The guide is only valid for Work.AI versions up to 10.2.9. Starting from v10.3, MinIO is replaced with S3 Manager application. For details on S3 Manager, read Organize and access objects in S3 Manager.

By default, Control Tower (CT) and VDS services use the same credentials for access to S3. It means that anyone with access to CT can get sensitive VDS data.

Let's assume that you have two teams working with the same CT instance, and you need to do the following:

• Eliminate the risk of incidental data sharing between the teams.
• Make sure that, during Business Process execution, each team retrieves their appropriate stored access and secret keys for the S3.

To meet the goals, create separate credentials and configure CT and VDS to use them. The configuration procedure is as described below.

Create bucket

Since each team should have its own bucket, create these buckets before you start the configuration. You can do this in the web interface, S3 browser, or console. If you use the console, run the following command:

minio-client mb {alias}/{bucket_name}

where:

• alias is the alias to the server configuration (by default, s3)
• bucket_name is the unique bucket name that you choose for the team

Configure Control Tower policy

Access to Control Tower should be limited access since each team can get CT keys from the Business Process context. Due to this limitation, the Control Tower policy will include the following permissions:

• Read/Write access to the vds-models bucket
• Write access to the vds-resources bucket

The vds-resources bucket is restricted to Write only as, during model training, it contains documents from the training set.

minio-client admin policy add {alias} {policy_name} tower_policy.json

where:

• alias is the alias to the server configuration (by default, s3).
• policy_name is the unique policy name that you choose for this case.

See sample tower_policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": [
        "s3:GetBucketLocation",
        "s3:PutObject",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": [
        "arn:aws:s3:::vds-resources/*"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::vds-models/*"
      ]
    }
  ]
}

Configure VDS service policy

The VDS service is responsible for training. Since it does not share keys with any resources, restrict this access just in case. The service will have the following permissions:

• Read/Write access to vds-models bucket
• Read/Write access to vds-resources bucket

minio-client admin policy add {alias} {policy_name} vdf_policy.json

where:

• alias is the alias to the server configuration (by default, s3).
• policy_name is the unique policy name that you choose for this case.

See sample vds_policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::vds-resources/*",
        "arn:aws:s3:::vds-models/*"
      ]
    }
  ]
}

Configure team service policy

Each team should have its own policy. Basically, teams need to have access to their personal buckets only. Depending on the team needs, it can be one or more buckets. The instruction shows a sample configuration where each team has access to one bucket only.

minio-client admin policy add {alias} {policy_name} {team_policy_file_name}.json

where:

• alias is the alias to the server configuration (by default, s3).
• policy_name is the unique policy name that you choose for this case.
• team_policy_file_name is the name of the team policy file.

See sample team_policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::team1-bucket/*"
      ]
    }
  ]
}

To check that all policies were created successfully, use the following command:

minio-client admin policy list {alias}

where:

• alias is the alias to the server configuration (by default, s3).

Create new keys (users)

To create new keys, follow the steps below:

1. Run the following command:

   minio-client admin user add {alias} {username} {password}

   where:

   • alias is the alias to the server configuration (by default, s3).
   • username is a unique username.
   • password is a unique password.

2. Assign a policy to the user:

   minio-client admin policy set {alias} {policy_name} user={access_key}

   where:

   • alias is the alias to the server configuration (by default, s3).
   • access_key is a new user access key.
   • policy_name is the name of the policy applied to users.

3. Check the results:

   minio-client admin user list {alias}

   where:

   • alias is the alias to the server configuration (by default, s3).

Configure DB server

To configure the database (DB) server, follow the steps below:

1. On the DB server, populate the INSTALL_DIR/secrets.yml file with the access and secret keys created for CT in the previous step.

2. To check that the INSTALL_DIR/distr/latest/secrets_overrides.yml and PACKAGE_DIR/VAULT_PASSWORD.DELETE_ME files exist, run the following command:

   cd PACKAGE_DIR
   ./install.sh passwords_update db -e previous_secrets_file=INSTALL_DIR/distr/latest/secrets_overrides.yml --vault-password-file=VAULT_PASSWORD.DELETE_ME

3. Restart DB services:

   wfmanager restart all

Finish configuration

To complete the configuration process, follow the steps below:

1. Copy secrets.yml, secrets_overrides.yml, and VAULT_PASSWORD.DELETE_ME to the Master server and run the following command:

   ./install.sh passwords_update app -e previous_secrets_file=<PATH>/secrets_overrides.yml --vault-password-file=<PATH>/VAULT_PASSWORD.DELETE_ME

2. Restart APP services:

   wfmanager restart all

3. On the AutoML Master, update the following files with the credentials for VDS you set when you created buckets.

   /opt/workfusion/minio_client_conf/config.json
   /opt/workfusion/s3cmd_conf/vds_s3cfg
   /opt/workfusion/vds-mesos-adapter/vds-mesos-adapter.properties
   /opt/workfusion/vds-scaling-service/vds-scaling-service.properties

4. Restart all services on AutoML Master.

   wfmanager restart all