Version: 10.3

Migrate S3 data to Work.AI v10.3

The guide describes how you can migrate data located in S3 when upgrading the platform from version 10.2.x to 10.3.

caution

After migrating to Ceph, the S3 URL changes from https://{host-name}-minio.cloud.workfusion.com to https://{host-name}-s3.cloud.workfusion.com. You must update all S3 URLs in your Business Processes accordingly as any full URLs using the old format will no longer work after migration.
As a workaround, when the old MinIO URLs are saved in your Data Stores or hardcoded in Business Processes, update the DNS entries to point to the legacy MinIO endpoints.

Meet prerequisites

The data migration procedure includes the following two stages:

Data backup: before the upgrade, all data is copied from S3 to a backup file storage. For that, you can leverage existing servers (like Master) or deploy a dedicated machine.
Restoration: it is the post-upgrade procedure when data is copied back from the backup storage to a newly deployed S3 service.

Before the upgrade procedure, meet the following prerequisites:

Provide a migration server (Linux OS) with network connectivity to the Master server.
Download a MinIO client binary to the directory where the script is executed. The Master server already has the binary downloaded. However, if you use a dedicated server, download the file from the official site and add it to the respective directory on the server.
Create a backup directory on the migration server with sufficient space to fit all files located in S3.

note

To estimate the size of files in S3, check the data in MinIO or run the minio-client du s3/ command from the Master server. Mind that executing the du command might take significant amount of time.

Prepare migration script

From the terminal on the migration server, download (or copy) the MinIO client binary to the current directory.

In the current directory, create a script with the migrate.sh name and the following contents:

#!/bin/bash

# S3 Backup/Restore Script
# Usage: ./s3_backup_restore.sh [backup|restore] [backup_dir]

extra_args="${@:3}"

set -e

# Configure s3-client connection params

S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_URL=

SOURCE_ALIAS=s3
DEST_ALIAS=s3_dest

# Check if s3-client is available
if ! command -v ./mc &> /dev/null; then
    echo "Error: s3-client is not installed or not in the expected path"
    echo "Please, place mc binary into the directory where script is executed"
    exit 1
fi

# Check arguments
if [ $# -le 1 ]; then
    echo "Usage: $0 [backup|restore] [backup_dir]"
    exit 1
fi

ACTION=$1
BACKUP_DIR=$2

# Check if action is valid
if [ "$ACTION" != "backup" ] && [ "$ACTION" != "restore" ]; then
    echo "Error: First argument must be either 'backup' or 'restore'"
    exit 1
fi

mkdir -p "$BACKUP_DIR"/data
mkdir -p "$BACKUP_DIR"/policies

# Calculate free space available for backup
get_available_space() {
    local dir="$1"
    # Get available space in KB
    local avail_kb=$(df -k "$dir" | awk 'NR==2 {print $4}')
    # Convert to GB
    echo $((avail_kb / 1024 / 1024))
}

# Prompt user before backup
prompt_before_backup() {
    echo "--- Available space in '$BACKUP_DIR': $(get_available_space $BACKUP_DIR) GB ---"
    echo "Please, verify that total size of files on s3 does not exceed this limit"
    echo "Validation can be performed either from MinIO UI, or by running command './mc du ${SOURCE_ALIAS}/'"
    echo "!!! Pay attention, that 'du' command may take significant amount of time"
    read -p "Do you want to proceed? (y/N): " confirm
    case "$confirm" in
        [yY][eE][sS]|[yY])
            echo "Proceeding..."
            ;;
        *)
            echo "Operation cancelled."
            exit 1
            ;;
    esac
}

# Prompt user before restore
prompt_before_restore() {
    echo "Please, verify that the size of the disk, where ceph is installed, is sufficient to accomodate files from $BACKUP_DIR"
    echo "The size of files in the $BACKUP_DIR could be checked by running command 'du -sh $BACKUP_DIR'"
    echo "!!! Pay attention, that 'du' command may take significant amount of time"
    read -p "Do you want to proceed? (y/N): " confirm
    case "$confirm" in
        [yY][eE][sS]|[yY])
            echo "Proceeding..."
           ;;
        *)
            echo "Operation cancelled."
            exit 1
            ;;
    esac
}

# Function to perform backup
perform_backup() {
  echo "Starting backup of S3 resources to $BACKUP_DIR..."

  # Using mirror command to backup all S3 data
  echo "Mirroring all S3 resources to backup directory $BACKUP_DIR ..."
  ./mc alias set ${SOURCE_ALIAS} https://$S3_URL ${S3_ACCESS_KEY} ${S3_SECRET_KEY}
  ./mc mirror ${SOURCE_ALIAS}/ "$BACKUP_DIR/data/"

  # Perform per-bucket backup of policies
  # Get a list of all bucket names from the source
  BUCKETS=$(./mc ls ${SOURCE_ALIAS} | awk '{print $NF}')

  # Loop through each bucket and save its policy
  for bucket in $BUCKETS; do
      echo "------------------------------------"
      echo "Processing bucket policy: $bucket"
      POLICY_FILE="${BACKUP_DIR}/policies/${bucket%/}.json"
      # Attempt to get the policy from the source bucket
      if ./mc anonymous  get-json ${SOURCE_ALIAS}/$bucket > "$POLICY_FILE"; then
      echo "  -> Policy for '$bucket' backed up to '$POLICY_FILE'."
      else
      echo "  -> No policy found for bucket '$bucket'. Skipping."
      fi
  done

  echo "Backup completed successfully!"
}

# Function to perform restore
perform_restore() {
  echo "Starting restore of S3 resources from $BACKUP_DIR..."

  # Using mirror command to backup all S3 resources
  echo "Mirroring all S3 resources from backup directory $BACKUP_DIR ..."
  ./mc alias set ${DEST_ALIAS} https://$S3_URL ${S3_ACCESS_KEY} ${S3_SECRET_KEY}
  ./mc mirror "$BACKUP_DIR/data/" ${DEST_ALIAS}/ --overwrite $extra_args

  for policy_file in "$BACKUP_DIR/policies/"*.json; do
      # Extract the bucket name from the filename (e.g., my-bucket.json -> my-bucket)
      bucket_name=$(basename "$policy_file" .json)
      echo "------------------------------------"
      echo "Restoring policy for bucket: $bucket_name"
      
      # Set the policy on the corresponding destination bucket
      if ./mc anonymous set-json "$policy_file" ${DEST_ALIAS}/$bucket_name; then
          echo "  -> Policy successfully restored to '$bucket_name'."
      else
          echo "  -> Failed to restore policy for '$bucket_name'."
          echo "  -> Check if the bucket exists on the destination and you have permissions."
      fi
  done

  echo "Restore completed successfully!"
}

# Main execution
if [ "$ACTION" = "backup" ]; then
    prompt_before_backup
    perform_backup
elif [ "$ACTION" = "restore" ]; then
    prompt_before_restore
    perform_restore
fi

exit 0

Set the ACCESS_KEY, SECRET_KEY, and URL parameters to the values from your current (10.2.x) environment. S3_URL should point to the MinIO lb hostname.
Make script executable:
```
chmod +x migrate.sh
```

Copy S3 data to backup directory

Make sure that there's a directory on your server that has enough space to fit all data from S3.
Back up the data from S3 by running the following command:
```
./migrate.sh backup <path to your backup directory>
```

The execution process should look similar to the following:

Starting backup of S3 resources to /opt/workfusion/s3_backup...
Mirroring all S3 resources to backup directory...
 1.31 GiB / 1.31 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 238.62 MiB/s 5s
Backup completed successfully!

Restore S3 data

Make sure that the disk on your server dedicated to the S3 storage has enough free space to fit all data from the backup.
Update the credentials in the script. Set the ACCESS_KEY, SECRET_KEY, and URL parameters to the values from your new (10.3.x) environment. S3_URL should point to the S3 lb hostname.
Restore the data from S3 by running the following command:
```
 ./migrate.sh restore <path to your backup directory>
```

Meet prerequisites​

Prepare migration script​

Copy S3 data to backup directory​

Restore S3 data​

Meet prerequisites

Prepare migration script

Copy S3 data to backup directory

Restore S3 data