9.2

Edit

Migration from Riak to MinIO

Overview

This guide is for migrating Riak, the object storage solution in SPA that was used in versions prior to 9.1, to Minio, the solution that has replaced Riak in SPA 9.1. The process for replacing Riak with Minio is simple. Install Minio, migrate the data, make sure permissions are correct after the data migration, make sure Minio services start when they're supposed to, and clean up the old Riak data directory.

To migrate the Riak data directory to the Minio data directory, you will essentially double the disk space used for object storage in your Workfusion volume. Please make sure there is enough disk space to replicate the Riak data directory on the same volume before proceeding.

Predefined common variables (set your own values in config.yml before installation):

  • /opt/workfusion - <INSTALL_DIR> - default Workfusion installation directory
  • /opt/workfusion/wf_installer - <PACKAGE_DIR> - directory where installation package was extracted

The list of S3 client apps are known to be compatible with Minio:

  • s3cmd 2.0.1
  • S3 Browser 7.4.5
  • Cyberduck 6.5.0

The list of S3 client apps that are incompatible with Minio:

  • s3cmd 1.1.0
  • DragonDisk 1.05

Minio installation

  1. Create Minio folders

    mkdir -p /opt/workfusion/minio/{data,etc}

  2. Get the Minio binary

    wget --no-check-certificate -c -O /opt/workfusion/minio/minio 'https://s3.amazonaws.com/workfusion-installer/blobs/binaries/minio/minio.RELEASE.2018-11-30T03-56-59Z'

  3. Make Minio binary executable

    chmod +x /opt/workfusion/minio/minio

  4. Create /opt/workfusion/minio/etc/config.json file with the following content. Replace 'accessKey' and 'secretKey' values with the real keys.

    {
"version": "23",
"credential": {
    "accessKey": "admin-key",
    "secretKey": "admin-secret"
}
}

    admin-key is your Riak accessKey

    admin-secret is your Riak secretKey

  5. Start Minio

    /opt/workfusion/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/workfusion/minio/etc /opt/workfusion/minio/data

    This command is useful to check that minio server is configured correctly and no errors during the startup appear. Control+C to stop it after you'll see

    [server@user etc]$ /opt/app/workfusion/apps/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/app/workfusion/apps/minio/etc /opt/app/workfusion/apps/minio/data

Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '23' to '24' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '24' to '25' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '25' to '26' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '26' to '27' successfully.
Configuration file /opt/app/workfusion/apps/minio/etc/config.json migrated from version '27' to '28' successfully.
Endpoint:  http://127.0.0.1:9000
AccessKey: *************************
SecretKey: **************************

Data Migration Steps

  1. Get the list of the buckets exists in riak.

    s3cmd ls

    Create folders inside the minio/data folder according to the list. Names should match the name of buckets.

    for i in `s3cmd ls | awk '{ print $3 }' | cut -d/ -f3` ; do mkdir /opt/app/workfusion/apps/minio/data/$i; done

  2. Download all the files from riak.

    for BUCKET in *; do s3cmd get s3://$BUCKET $BUCKET -r; done

  3. Download minio client:

    cd /opt/workfusion/tools
wget -O 'mc' 'https://workfusion-installer.s3.amazonaws.com/blobs/binaries/minio/mc.RELEASE.2018-11-06T01-12-20Z'

  4. Make it executable:

    chmod +x mc

    Create directory for minio config

    mkdir <INSTALL_DIR>/minio_client_conf/

  5. Rename /opt/workfusion/nginx/upstream/riak.conf file to s3.conf

    mv /opt/workfusion/nginx/upstream/riak.conf /opt/workfusion/nginx/upstream/s3.conf

    Edit upstream/s3.conf and change port 8081 to 9000

  6. Rename /opt/workfusion/nginx/db_apps.d/riak.conf file to s3.conf

    mv /opt/workfusion/nginx/db_apps.d/riak.conf /opt/workfusion/nginx/db_apps.d/s3.conf

    Edit db_apps.d/s3.conf and rename "riak-cs" location to "minio":

    location /minio {
proxy_pass http://s3;
}

  7. Reload nginx

    wfmanager restart nginx

    For old versions you may want to use systemctl command

WFManager approach (8.5+)

  1. Create new minio.ini config file under <INSTALL_DIR>/supervisord/apps/.

    [program:minio]
command = bash -c '
source /opt/workfusion/environment.sh
/opt/workfusion/minio/minio server /opt/workfusion/minio/data --config-dir /opt/workfusion/minio/etc --address "127.0.0.1:9000"
'
stopasgroup=true
autostart = true

stdout_logfile = /opt/workfusion/supervisord/log/%(program_name)s.out.log
stderr_logfile = /opt/workfusion/supervisord/log/%(program_name)s.err.log

  2. Update wfmanager to find new config

    wfmanager update

  3. Run minio

    wfmanager start minio

Systemd systemwide approach (pre 8.5)

  1. Put the following into /etc/systemd/system/minio.service

    [Unit]
Description=Minio
Documentation=https://docs.minio.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/workfusion/minio/minio

[Service]
WorkingDirectory=/opt/workfusion/minio

User=wfuser
Group=wfuser

PermissionsStartOnly=true

EnvironmentFile=-/opt/workfusion/minio/etc/minio
ExecStartPre=/bin/bash -c "[ -n \"${MINIO_VOLUMES}\" ] || echo 'Variable MINIO_VOLUMES not set in /opt/workfusion/minio/etc/minio'"

ExecStart=/opt/workfusion/minio/minio server $MINIO_OPTS $MINIO_VOLUMES

## Let systemd restart this service only if it has ended with the clean exit code or signal.
Restart=on-success

StandardOutput=journal
StandardError=inherit

## Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

## Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

## SIGTERM signal is used to stop Minio
KillSignal=SIGTERM

SendSIGKILL=no

SuccessExitStatus=0

[Install]
WantedBy=multi-user.target

    Original systemd config file: https://github.com/minio/minio-service/blob/master/linux-systemd/minio.service

  2. Put the following into /opt/workfusion/minio/etc/minio

    MINIO_VOLUMES=/opt/workfusion/minio/data
MINIO_OPTS="-C /opt/workfusion/minio/etc --address 127.0.0.1:9000"

  3. Issue the following commands

    systemctl daemon-reload
systemctl enable minio
systemctl restart minio

Set permissions to public files

  1. Add to minio new server

    ./mc -C <INSTALL_DIR>/minio_client_conf/ config host add minio http://localhost:9000 <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> s3v4 --insecure

  2. Run next command to set all buckets to public:

    for i in $(ls /opt/workfusion/minio/data); do /opt/workfusion/minio/mc policy --recursive public minio/$i; done

    Assuming $MINIO_PATH is where minio is installed

    If you have at least one bucket that should be restricted use the next command:

    /opt/workfusion/minio/mc policy public minio/bucket_name

    The list of buckets that should be public:

    • cv-help
    • workspace-resources
    • workspace
    • workfusiondevelopment
    • workfusion-resources
    • workfusiondevelopment
    • tinymce-resources

Riak cleanup process

  1. Clear Riak buckets data.

    Execute the following command bellow:

    cd <INSTALL_DIR>/tools
./mc -C <INSTALL_DIR>/minio_client_conf/ rm --recursive --force --dangerous riak

    The similar output should appear:

    Removing `riak/doc-upload/260a5e68-617e-4720-a7cd-56183cd88223.pdf`.
Removing `riak/doc-upload/296e3683-f63e-4b25-81f4-484cee204e57.pdf`.
Removing `riak/doc-upload/3a74fbaa-f9b5-41f4-b35a-824da55f8e50.html`.
Removing `riak/doc-upload/46166147-3d95-4e6c-b73d-e3e2d3d3b071.html`.
Removing `riak/doc-upload/b7322349-5988-454e-b2f0-767b7759f5e5.html`.
Removing `riak/doc-upload/bf48afff-2578-4bc4-881d-e8c1ce43d303.pdf`.
Removing `riak/doc-upload/e009b010-898f-42a8-a97d-659c43fb1d42.html`.
Removing `riak/doc-upload/f34ee6eb-9aba-466c-8c84-4bec6ee4d36d.pdf`.
Removing `riak/doc-upload`.
Removing `riak/recorder-images`.
Removing `riak/secdocs`.
Removing `riak/tinymce-resources`.

  2. Stop all Riak components.

    cd <INSTALL_DIR>/
wfmanager stop riak-stanchion
wfmanager stop riak
wfmanager stop riak-cs

  3. Delete init scripts for Riak.

    cd <INSTALL_DIR>/supervisord/apps
rm riak.ini
rm riak-cs.ini
rm riak-stanchion.ini

  4. Delete Riak directories.

    cd <INSTALL_DIR>/
rm -rf riak
rm riak-cs
rm riak-stanchion

Monitoring and WFManager cleanup for APM

  1. Go to <INSTALL_DIR>/supervisord/apps/ (e.g. /opt/wotkfusion/supervisord/apps/) and move riak files to another place:

    cd /opt/wotkfusion/supervisord/apps/
mv riak* /opt/workfusion

  2. Update wfmanager configuration

    wfmanager update

    Riak services should no longer be observed on wfmanager status output

  3. On DB server, go to <INSTALL_DIR>/wfagent/settings/ (e.g. /opt/wotkfusion/wfagent/settings/) and move riak files to another place:

    cd /opt/wotkfusion/ wfagent/settings/
mv riak* /opt/workfusion

  4. Copy attached s3.yml file to /opt/wotkfusion/wfagent/settings/

    s3.yml

    environment:
minio_hostname: 127.0.0.1
minio_listen_port: 9000

modules.minio:

components:
    - { id: minio, name: "File Storage (Minio)", group: db, alerts: [ minio ] }

commands:
    - id: restart.minio
    type: command
    proxy: true
    expression: "/opt/app/workfusion/python-site/bin/wfmanager restart minio"

datasources:
    - { id: Minio, type: http, protocol: http, host: "%minio_hostname%", port: "%minio_listen_port%" }
    - { id: MinioTcp, type: tcp, host: "%minio_hostname%", port: "%minio_listen_port%" }


alerts:
    - id: minio
    triggers:
        - { metric: minio.tcp.status, neq: 0, message: "Problem with riak.tcp check" }
        - { metric: minio.tcp.listening, neq: 1, message: "No TCP port is listening" }
        - { metric: minio.stats.status, neq: 0, message: "Web check failed" }
        - { metric: minio.stats.http.code, neq: 200, message: "Bad response code" }


checks:
    - id: minio.stats
    type: http
    expression: "/minio/health/ready"
    source: Minio

    - id: minio.tcp
    type: tcp
    source: MinioTcp

  5. Restart wfagent service on DB server

    wfmanager restart wfagent

  6. Restart wfagent service on APM server

    wfmanager restart wfagent

  7. Check APM WEB UI

Example session from live customer deployment

## DB server steps
 
/opt/app/workfusion/apps/
mkdir -p /opt/app/workfusion/apps/minio/{data,etc}
 
systemctl daemon-reload
systemctl enable minio
systemctl restart minio
 
wget --no-check-certificate -c -P /opt/app/workfusion/apps/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
 
/opt/app/workfusion/apps/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/app/workfusion/apps/minio/etc /opt/app/workfusion/apps/minio/data
 
## Script to create the S3 folders locally
 
for i in `s3cmd ls | awk '{ print $3 }' | cut -d/ -f3` ; do mkdir /opt/app/workfusion/apps/minio/data/$i; done
 
## Migrate all data
 
## To copy the S3 content to local content (1 hr)
 
for BUCKET in *; do s3cmd get s3://$BUCKET $BUCKET -r >> output; done
 
## Compare results with
 
du -sch *
 
wget https://dl.minio.io/client/mc/release/linux-amd64/mc
chmod +x mc
 
## Permissions
 
$ pwd
/opt/app/workfusion/apps/minio
$ for i in $(ls /opt/app/workfusion/apps/minio/data); do /opt/app/workfusion/apps/minio/mc policy public ./data/$i; done
 
$ cd /opt/app/workfusion/apps/minio/etc
$ cat minio
MINIO_VOLUMES=/opt/app/workfusion/apps/minio/data
MINIO_OPTS="-C /opt/app/workfusion/apps/minio/etc --address 0.0.0.0:9000"
 
./mc -C /opt/app/workfusion/apps/minio/minio_client_conf/ config host add minio http://localhost:9000 Access_Key Secret_key s3v4 --insecure
./mc ls minio
 
## App Server steps
 
$ pwd
/opt/app/workfusion/apps/nginx/etc/upstream
$ cat s3-standalone
upstream s3 {
    server FQDN:9000;
}
 
## Stop Riak Service (DB server)
 
$ riak-cs-service stop
$ stanchion-service stop
$ riak-service stop

## Disable riak services from starting at boot

$ systemctl disable riak-cs-service 
$ systemctl disable stanchion-service 
$ systemctl disable riak-service

## Restart ngnix (App server)
## Restart minio (DB server)
← PreviousNext →