9.4

Edit

Migration from Riak to MinIO

Overview

This guide is for migrating Riak, the object storage solution in SPA that was used in versions prior to 9.1, to MinIO, the solution that has replaced Riak in SPA 9.1. The process for replacing Riak with MinIO is simple. Install MinIO, migrate the data, make sure permissions are correct after the data migration, make sure MinIO services start when they're supposed to, and clean up the old Riak data directory.

To migrate the Riak data directory to the MinIO data directory, you will essentially double the disk space used for object storage in your WorkFusion volume. Make sure there is enough disk space to replicate the Riak data directory on the same volume before proceeding.

Predefined common variables (set your own values in config.yml before installation):

  • /opt/workfusion: default Workfusion installation directory
  • /opt/workfusion/wf_installer: directory where the installation package was extracted The list of S3 client apps are known to be compatible with MinIO:
  • s3cmd 2.0.1
  • S3 Browser 7.4.5
  • Cyberduck 6.5.0

The list of S3 client apps that are incompatible with MinIO:

  • s3cmd 1.1.0
  • DragonDisk 1.05

MinIO installation

  1. Create MinIO folders:
mkdir -p /opt/workfusion/minio/{data,etc}
  1. Get the MinIO binary:
wget --no-check-certificate -c -O /opt/workfusion/minio/minio 'https://s3.amazonaws.com/workfusion-installer/blobs/binaries/minio/minio.RELEASE.2018-11-30T03-56-59Z'
  1. Make MinIO binary executable:
chmod +x /opt/workfusion/minio/minio
  1. Create the /opt/workfusion/minio/etc/config.json file with the following content. Replace the accessKey and secretKey values with the real keys.
{
  "version": "23",
  "credential": {
    "accessKey": "admin-key",
    "secretKey": "admin-secret"
  }
}

Where: admin-key is your Riak accessKey. admin-secret is your Riak secretKey.

  1. Start MinIO:
/opt/workfusion/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/workfusion/minio/etc /opt/workfusion/minio/data

This command is useful to check that the MinIO server is configured correctly and no errors during the startup appear. Control+C to stop it after you'll see

[server@user etc]$ /opt/app/workfusion/apps/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/app/workfusion/apps/minio/etc /opt/app/workfusion/apps/minio/data

Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '23' to '24' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '24' to '25' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '25' to '26' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '26' to '27' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '27' to '28' successfully.
Endpoint:  http://127.0.0.1:9000
AccessKey: *************************
SecretKey: **************************

Data migration steps

  1. Get the list of the buckets exists in riak.
s3cmd ls
  1. Create folders inside the minio/data folder according to the list. Names should match the name of buckets.
for i in `s3cmd ls | awk '{ print $3 }' | cut -d/ -f3` ; do mkdir /opt/app/workfusion/apps/minio/data/$i; done
  1. Download all the files from riak:
for BUCKET in *; do s3cmd get s3://$BUCKET $BUCKET -r; done
  1. Download the MinIO client:
cd /opt/workfusion/tools
wget -O 'mc' 'https://workfusion-installer.s3.amazonaws.com/blobs/binaries/minio/mc.RELEASE.2018-11-06T01-12-20Z'
  1. Make it executable:
chmod +x mc
  1. Create a directory for the MinIO config:
mkdir <INSTALL_DIR>/minio_client_conf/
  1. Rename the /opt/workfusion/nginx/upstream/riak.conf file to s3.conf.
mv /opt/workfusion/nginx/upstream/riak.conf /opt/workfusion/nginx/upstream/s3.conf

  1. Edit upstream/s3.conf and change port 8081 to 9000.

  2. Rename the /opt/workfusion/nginx/db_apps.d/riak.conf file to s3.conf.

mv /opt/workfusion/nginx/db_apps.d/riak.conf /opt/workfusion/nginx/db_apps.d/s3.conf
  1. Edit db_apps.d/s3.conf and rename the riak-cs location to minio:
location /minio {
 proxy_pass http://s3;
}
  1. Reload Nginx:
wfmanager restart nginx

For old versions, you can use the systemctl command.

WFManager approach (8.5+)

  1. Create a new minio.ini config file under <INSTALL_DIR>/supervisord/apps/.
[program:minio]
command = bash -c '
 source /opt/workfusion/environment.sh
 /opt/workfusion/minio/minio server /opt/workfusion/minio/data --config-dir /opt/workfusion/minio/etc --address "127.0.0.1:9000"
 '
stopasgroup=true
autostart = true

stdout_logfile = /opt/workfusion/supervisord/log/%(program_name)s.out.log
stderr_logfile = /opt/workfusion/supervisord/log/%(program_name)s.err.log
  1. Update wfmanager to find a new config:
wfmanager update
  1. Run MinIO:
wfmanager start minio

Systemd systemwide approach (pre 8.5)

  1. Put the following into /etc/systemd/system/minio.service:
[Unit]
Description=Minio
Documentation=https://docs.minio.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/workfusion/minio/minio

[Service]
WorkingDirectory=/opt/workfusion/minio

User=wfuser
Group=wfuser

PermissionsStartOnly=true

EnvironmentFile=-/opt/workfusion/minio/etc/minio
ExecStartPre=/bin/bash -c "[ -n \"${MINIO_VOLUMES}\" ] || echo 'Variable MINIO_VOLUMES not set in /opt/workfusion/minio/etc/minio'"

ExecStart=/opt/workfusion/minio/minio server $MINIO_OPTS $MINIO_VOLUMES

## Let systemd restart this service only if it has ended with the clean exit code or signal.
Restart=on-success

StandardOutput=journal
StandardError=inherit

## Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

## Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

## SIGTERM signal is used to stop Minio
KillSignal=SIGTERM

SendSIGKILL=no

SuccessExitStatus=0

[Install]
WantedBy=multi-user.target

The original systemd config file: https://github.com/minio/minio-service/blob/master/linux-systemd/minio.service

  1. Put the following into /opt/workfusion/minio/etc/minio:
MINIO_VOLUMES=/opt/workfusion/minio/data
MINIO_OPTS="-C /opt/workfusion/minio/etc --address 127.0.0.1:9000"
  1. Issue the following commands:
systemctl daemon-reload
systemctl enable minio
systemctl restart minio

Set permissions to public files

  1. Add a new server to MinIO:
./mc -C <INSTALL_DIR>/minio_client_conf/ config host add minio http://localhost:9000 <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> s3v4 --insecure
  1. Run the next command to set all buckets to public:
for i in $(ls /opt/workfusion/minio/data); do /opt/workfusion/minio/mc policy --recursive public minio/$i; done

Assuming $MINIO_PATH is where MinIO is installed.

If you have at least one bucket that should be restricted, use the next command:

/opt/workfusion/minio/mc policy public minio/bucket_name

The list of buckets that should be public:

  • cv-help
  • workspace-resources
  • workspace
  • workfusiondevelopment
  • workfusion-resources
  • workfusiondevelopment
  • tinymce-resources

Riak cleanup process

  1. Clear the Riak buckets data.

  2. Execute the following command:

cd <INSTALL_DIR>/tools
./mc -C <INSTALL_DIR>/minio_client_conf/ rm --recursive --force --dangerous riak

The similar output should appear:

Removing `riak/doc-upload/260a5e68-617e-4720-a7cd-56183cd88223.pdf`.
Removing `riak/doc-upload/296e3683-f63e-4b25-81f4-484cee204e57.pdf`.
Removing `riak/doc-upload/3a74fbaa-f9b5-41f4-b35a-824da55f8e50.html`.
Removing `riak/doc-upload/46166147-3d95-4e6c-b73d-e3e2d3d3b071.html`.
Removing `riak/doc-upload/b7322349-5988-454e-b2f0-767b7759f5e5.html`.
Removing `riak/doc-upload/bf48afff-2578-4bc4-881d-e8c1ce43d303.pdf`.
Removing `riak/doc-upload/e009b010-898f-42a8-a97d-659c43fb1d42.html`.
Removing `riak/doc-upload/f34ee6eb-9aba-466c-8c84-4bec6ee4d36d.pdf`.
Removing `riak/doc-upload`.
Removing `riak/recorder-images`.
Removing `riak/secdocs`.
Removing `riak/tinymce-resources`.
  1. Stop all Riak components.
cd <INSTALL_DIR>/
wfmanager stop riak-stanchion
wfmanager stop riak
wfmanager stop riak-cs
  1. Delete init scripts for Riak.
cd <INSTALL_DIR>/supervisord/apps
rm riak.ini
rm riak-cs.ini
rm riak-stanchion.ini
  1. Delete Riak directories.
cd <INSTALL_DIR>/
rm -rf riak
rm riak-cs
rm riak-stanchion

Monitoring and WFManager cleanup for APM

  1. Go to <INSTALL_DIR>/supervisord/apps/ (for example, /opt/wotkfusion/supervisord/apps/) and move riak files to another place:
cd /opt/wotkfusion/supervisord/apps/
mv riak* /opt/workfusion
  1. Update the wfmanager configuration:
wfmanager update

Riak services should no longer be observed on wfmanager status output

  1. On DB server, go to <INSTALL_DIR>/wfagent/settings/ (for example, /opt/wotkfusion/wfagent/settings/) and move riak files to another place:
cd /opt/wotkfusion/ wfagent/settings/
mv riak* /opt/workfusion
  1. Copy the attached s3.yml file to /opt/wotkfusion/wfagent/settings/.

s3.yml:

environment:
  minio_hostname: 127.0.0.1
  minio_listen_port: 9000

modules.minio:

  components:
    - { id: minio, name: "File Storage (Minio)", group: db, alerts: [ minio ] }

  commands:
    - id: restart.minio
      type: command
      proxy: true
      expression: "/opt/app/workfusion/python-site/bin/wfmanager restart minio"

  datasources:
    - { id: Minio, type: http, protocol: http, host: "%minio_hostname%", port: "%minio_listen_port%" }
    - { id: MinioTcp, type: tcp, host: "%minio_hostname%", port: "%minio_listen_port%" }


  alerts:
    - id: minio
      triggers:
        - { metric: minio.tcp.status, neq: 0, message: "Problem with riak.tcp check" }
        - { metric: minio.tcp.listening, neq: 1, message: "No TCP port is listening" }
        - { metric: minio.stats.status, neq: 0, message: "Web check failed" }
        - { metric: minio.stats.http.code, neq: 200, message: "Bad response code" }


  checks:
    - id: minio.stats
      type: http
      expression: "/minio/health/ready"
      source: Minio

    - id: minio.tcp
      type: tcp
      source: MinioTcp
  1. Restart the wfagent service on DB server:
wfmanager restart wfagent
  1. Restart the wfagent service on APM server:
wfmanager restart wfagent
  1. Check the APM WEB UI.

Example session from live customer deployment

Example session:

## DB server steps
 
/opt/app/workfusion/apps/
mkdir -p /opt/app/workfusion/apps/minio/{data,etc}
 
systemctl daemon-reload
systemctl enable minio
systemctl restart minio
 
wget --no-check-certificate -c -P /opt/app/workfusion/apps/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
 
/opt/app/workfusion/apps/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/app/workfusion/apps/minio/etc /opt/app/workfusion/apps/minio/data
 
## Script to create the S3 folders locally
 
for i in `s3cmd ls | awk '{ print $3 }' | cut -d/ -f3` ; do mkdir /opt/app/workfusion/apps/minio/data/$i; done
 
## Migrate all data
 
## To copy the S3 content to local content (1 hr)
 
for BUCKET in *; do s3cmd get s3://$BUCKET $BUCKET -r >> output; done
 
## Compare results with
 
du -sch *
 
wget https://dl.minio.io/client/mc/release/linux-amd64/mc
chmod +x mc
 
## Permissions
 
$ pwd
/opt/app/workfusion/apps/minio
$ for i in $(ls /opt/app/workfusion/apps/minio/data); do /opt/app/workfusion/apps/minio/mc policy public ./data/$i; done
 
$ cd /opt/app/workfusion/apps/minio/etc
$ cat minio
MINIO_VOLUMES=/opt/app/workfusion/apps/minio/data
MINIO_OPTS="-C /opt/app/workfusion/apps/minio/etc --address 0.0.0.0:9000"
 
./mc -C /opt/app/workfusion/apps/minio/minio_client_conf/ config host add minio http://localhost:9000 Access_Key Secret_key s3v4 --insecure
./mc ls minio
 
## App Server steps
 
$ pwd
/opt/app/workfusion/apps/nginx/etc/upstream
$ cat s3-standalone
upstream s3 {
    server FQDN:9000;
}
 
## Stop Riak Service (DB server)
 
$ riak-cs-service stop
$ stanchion-service stop
$ riak-service stop

## Disable riak services from starting at boot

$ systemctl disable riak-cs-service 
$ systemctl disable stanchion-service 
$ systemctl disable riak-service

## Restart ngnix (App server)
## Restart minio (DB server)
← PreviousNext →