Version: 10.3.1

Set up disaster recovery

Disaster recovery (DR) is the ability to continue services' operation in case of major outages, often with reduced capabilities or performance. Disaster recovery solutions typically involve manual activities. DR is aimed at minimal data loss, including up to 15 seconds for database loss, up to 15 minutes for file system data loss, and up to 4 hours of server downtime.

The DR architecture implies having two sites: primary and DR one. The primary set hosts the complete set of servers (either in the HA mode or without it) with running components. The DR site is the exact copy of the primary one, including the servers' DNS and topology. All the servers on the DR site are running while the components on them are stopped.

The replication of data between the sites is performed every 30 minutes. If the primary site fails, the system is switched to the DR site with minimum data loss.

Requirements

The INSTALL_DIR/shared shared directory must be mounted to the Master servers both on the primary and the DR sites.
The shared NFS directory must be replicated between the primary and the DR sites.
MS SQL must support the Log Shipment and the Asynchronous Replication methods.
The client must configure the Global DNS server or Route 53 to enable site failover.
The DR site must mirror the following objects of the primary site:
- Servers' topology
- DNS records
- Credentials for services
All the Work.AI components must be stopped on the DR site.

Deploy DR environment

Prerequisites

The primary environment is deployed as usual and stopped.
NFS directories from the primary site are replicated to the DR site's NFS.
MSSQL is replicated from the primary site to the DR one.
DNS records point to the DR environment.

Deployment

To deploy the Disaster Recovery environment, do the following:

Copy INSTALLER_DIR (for example, /opt/workfusion/wf_installer) from the primary site (the MASTER1 server) to the DR one.
note
If you transfer the installer directory with rsync, make sure to add the -l, --links copy symlinks as symlinks option.
Install the Product to the DR environment as usual while skipping the RPA and BI setup:
```
$ cd $INSTALLER_DIR
$ ./install.sh install full -e skip_rpa=true -e skip_bi=true
```
Stop all services.
- Standard setup
- Multipoint setup
For the Standard installation, run the commands:
$ cd ${INSTALLER_DIR} $ ./install.sh stop_services full -e skip_rpa=true -e skip_bi=true
For the Multipoint installation:
- On each Master server, run the commands:
  # this line could be executed only once $ wfmanager stop marathon-apps $ wfmanager stop all
- On each Agent server, run the command:
  $ wfmanager stop all
Copy ${INSTALL_DIR}/vds-data/ssl/marathon.jks from primary environment to the DR environment, replacing the existing one
Restore etcd, ZooKeeper, and Elasticsearch data:
1. To restore etcd on each Master server, run the command:
```
$ etcd-restore.sh ${install_dir}/shared/etcd/backups/etcd_snapshot_<...>.db 
# Path to correct backup file, produced by primary environment
```
  note
  Remember to specify the path to a backup file.
  Otherwise, the restoration script automatically uses the latest backup in the /backups directory. Thus, it can accidentally use the file produced by the cron backup script in an incorrect environment.
2. To restore ZooKeeper on each Master server, run the command:
```
$ zookeeper-restore.sh ${install_dir}/shared/zookeeper/backups/zk-backup-<...>.zip 
# Path to correct backup file, produced by primary environment
```
  note
  Remember to specify the path to a backup file.
  Otherwise, the restoration script automatically uses the latest backup in the /backups directory. Thus, it can accidentally use the file produced by the cron backup script in an incorrect environment.
3. To restore Elasticsearch:
  1. Start the Elasticsearch cluster by running the following command on each Master server:
    $ wfmanager start elasticsearch
  2. Restore Elasticsearch security indices by running the following command once on any Master server:
    $ elasticsearch-restore.sh

Start all services.

Standard setup
Multipoint setup

For the Standard installation, run the commands:

$ cd ${INSTALLER_DIR}
$ ./install.sh start_services full -e skip_rpa=true -e skip_bi=true

For the Multipoint installation:

On each Master server, run the commands:

$ wfmanager start all
# the following line can be executed only once
$ wfmanager start marathon-apps 

On each Agent server, run the commands:
```
$ wfmanager start all
```

Install RPA and BI components:

$ ./install.sh install rpa
$ ./install.sh install bi

Verify that the deployed environment is working:

$ ./install.sh check full -e test_ml=true -e test_ocr=true -e test_rpa=true -e test_desktop=true -e test_ie=true

After setting the DR site, remember to disable the autostart of its processes. For that, on the DR site, perform the following operations:
1. On all Linux servers, run the following command to stop the processes. The processes on other servers must be stopped manually.
```
$ wfmanager stop marathon-apps
$ wfmanager stop all
```
2. On each server, run the following command to rename the directory with the startup scripts:
```
$ mv ${INSTALL_DIR}/supervisord/apps ${INSTALL_DIR}/supervisord/apps_disabled
$ wfmanager update
```
Switch the DNS records to point back to the primary site.

Configure DR failover

Switch from Primary site to DR

Failover is switching from a primary site to the DR one.

To switch to the DR site, do the following:

Stop all processes (for example, CT, workfusion, and so on) on the primary site.
It is recommended to shut down all servers with running processes. Alternatively, you can use the command wfmanager stop marathon-apps && wfmanager stop all to halt the processes on the Linux servers. The processes on other servers, including MS SQL, must be stopped manually. For example, to stop the Analytics service on the BI server, run the following commands:
```
$ tsm login
$ tsm stop
```
Switch the DNS records and IP addresses from the primary site to the DR one.
Perform the failover procedure for MS SQL according to your standard process.

On each Linux server, restore supervisord configuration:

$ mv ${INSTALL_DIR}/supervisord/apps_disabled ${INSTALL_DIR}/supervisord/apps
$ wfmanager update
$ wfmanager stop all

Restore etcd and ZooKeeper data. See step 7 in the Deployment section.
Start all services:
1. On the DR site, on each Linux server, start all processes:
```
$ wfmanager start all
```
2. On the DR site on any Master server, run the command:
```
$ wfmanager start marathon-apps
```
3. The processes on the Windows servers, including MS SQL, are to be started via the user interface or with specific initiation commands. For example, to run the Analytics service on the BI server, run the following commands:
```
$ tsm login
$ tsm start
```

After the replication, S3 and Nexus are immediately updated to the latest state, as they both are installed in a shared directory.

Switch from DR to primary site

Once an incident is resolved, you may want to switch from the DR site back to the primary one or to a new environment.

On the DR site

To switch from the DR site to the primary one, on the Disaster Recovery site, perform the following operations:

On each Master server, run the etcd backup procedure:
```
$ etcd-backup.sh
```
On each Master server, run the ZooKeeper backup procedure:
```
$ zookeeper-backup.sh
```
Stop the processes on the Linux servers:
```
$ wfmanager stop marathon-apps
$ wfmanager stop all
```
The processes on other servers, including MS SQL, must be stopped manually. For example, to stop the Analytics service on the BI server, run the following commands:
```
$ tsm login
$ tsm stop
```

On Linux servers, disable services autostart by renaming the directory with startup scripts:

$ mv ${INSTALL_DIR}/supervisord/apps ${INSTALL_DIR}/supervisord/apps_disabled
$ wfmanager update

Optional. Stop all DR servers.
Switch the DNS records and IP addresses from the DR site to the primary one.
Perform the failover procedure for MS SQL.

On the primary site

Continue the procedure and, on the primary site, perform the following operations:

Replicate common_nfs_share from the DR site.
Start all Master servers.

Stop all services on Master servers.

$ wfmanager stop marathon-apps
$ wfmanager stop all

Restore etcd and ZooKeeper data:
1. To restore etcd on each Master server, run the command:
```
$ etcd-restore.sh ${install_dir}/shared/etcd/backups/etcd_snapshot_<...>.db 
# Path to correct backup file, produced by dr environment
```
  note
  Remember to specify the path to a backup file.
  Otherwise, the restoration script automatically uses the latest backup in the /backups directory. Thus, it can accidentally use the file produced by the cron backup script in an incorrect environment.
2. To restore ZooKeeper, on each Master server, run the command:
```
$ zookeeper-restore.sh ${install_dir}/shared/zookeeper/backups/zk-backup-<...>.zip 
# Path to correct backup file, produced by dr environment
```
  note
  Remember to specify the path to a backup file.
  Otherwise, the restoration script automatically uses the latest backup in the /backups directory. Thus, it can accidentally use the file produced by the cron backup script in an incorrect environment.

Start all processes on Master servers.

$ wfmanager start all
$ wfmanager start marathon-apps

Start other Linux and Windows servers.
The processes on the Windows servers, including MS SQL, are to be started via UI or with specific initiation commands. For example, to run the Analytics service on the BI server, run the following commands:
```
$ tsm login
$ tsm start
```
Run a health check Business Process to verify that the system is restored correctly.

Recover OCR tasks

You can recover OCR tasks that may not be processed in case of disaster on the entire platform.

You can start the recovery process on one OCR server (node) at a time.

Start recovery

The REST endpoint /api/v2/recovery allows starting recovery immediately and gives a chance to restore tasks stuck in the IN_PROGRESS or QUEUED status.

There are two ways to initiate recovery:

Restart the OCR REST process manually. Recovery is started automatically once the app is up.
Send the POST request to the /api/v2/recovery OCR API endpoint. Recovery is started on one of the OCR servers in a cluster.

API messages

Message	Recovery state
Recovery is in progress	Recovery has started successfully
No tasks for recovery	Recovery process hasn't found any task for recovery
Recovery start was skipped because it had started by another service	User tries to start recovery when recovery is already in progress. Recovery request is skipped.
Recovery start is failed	Recovery cannot be started for some reason (for example, problems with connection to ZooKeeper)

Recovery process

recovery_status was introduced for a task as a database column to implement the recovery procedure. It is returned with general task info in the /getTaskStatus API.

Once the recovery process is started, OCR looks in the database to gather information about tasks with the InProgress, Queued, Submitted status.

Task state before recovery Task state after recovery

Submitted Submitted tasks cannot be recovered. OCR recovery marks their status as ProcessingFailed and sets RecoveryStatus to Skipped.

Task state before recovery	Task state after recovery
Submitted	Submitted tasks cannot be recovered. OCR recovery marks their status as `ProcessingFailed` and sets `RecoveryStatus` to `Skipped`.
Queued and InProgress	For Queued and InProgress tasks, the recovery process checks task data for consistency to ensure that recovery is meaningful for this particular task. Data consistency is checked between MSSQL and S3. If inconsistent task data (image files or pattern) is missing in S3 or MSSQL, a task is marked with the `ProcessingFailed` status, and the recovery status is set to `Skipped`. If data is consistent between S3 and MSSQL, the recovery process sends the task to processing, so the task is pushed to RabbitMQ and waits for its turn for processing. The recovery status is set to `Success`.

Queued and InProgress

For Queued and InProgress tasks, the recovery process checks task data for consistency to ensure that recovery is meaningful for this particular task. Data consistency is checked between MSSQL and S3. If inconsistent task data (image files or pattern) is missing in S3 or MSSQL, a task is marked with the ProcessingFailed status, and the recovery status is set to Skipped.

If data is consistent between S3 and MSSQL, the recovery process sends the task to processing, so the task is pushed to RabbitMQ and waits for its turn for processing. The recovery status is set to Success.

Sync S3 content from primary to DR site

Extract the rclone binary. It's available in the installer's sources/ directory.

$ mkdir -pv ${install_dir}/rclone
$ unzip ${install_dir}/wf_installer/sources/rclone-v1.71.1-linux-amd64.zip -d ${install_dir}/rclone/

Navigate to the rclone-v1.71.1-linux-amd64 directory. You'll find the rclone binary inside.
```
$ cd /opt/workfusion/rclone/rclone-v1.71.1-linux-amd64
$ ls -la
```
Create the rclone.conf file here with the following content:

[ceph-primary]
type = s3
provider = Ceph
access_key_id = XXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXX
endpoint = https://primary_s3_lb_hostname

[ceph-dr]
type = s3
provider = Ceph
access_key_id = XXXXXXXXXXXX
secret_access_key = XXXXXXXXXXXXXXXXXXXX
endpoint = http://127.0.0.1:7480

Make sure to replace the primary_s3_lb_hostname variable placeholder with an actual value. Also, fill the correct values for the access_key_id and secret_access_key variables for the primary and DR environments. They can be taken from config.yml and hosts.yml.

note

In this case, for ceph-dr, we use the local http://127.0.0.1:7480 rados-gateway endpoint so that we don't have to pass the requests to the local Ceph via the Load Balancer -> HAProxy -> Nginx route. We'll use direct API for faster write operations. 7480 is a local port where the Ceph rados-gateway is listening on the Master server.

Synchronize the entire S3 content from the primary environment to DR.

$ ./rclone sync ceph-primary: ceph-dr: --progress --stats=30s --create-empty-src-dirs

note

The synchronization will make the DR S3 content IDENTICAL to the primary environment. It means that if any files were present on the DR site but absent on the primary one, they will be removed from the DR site. In other words, it will make the Ceph content on the DR site an EXACT copy of the primary one.

Repeat the rclone command from the previous step periodically (for example, daily or once in several hours). It will synchronize the delta between the previous and current executions. To run it automatically in the desired time at desired intervals, use a supervisor-based cron or a standard Linux cron job.

Requirements​

Deploy DR environment​

Prerequisites​

Deployment​

Configure DR failover​

Switch from Primary site to DR​

Switch from DR to primary site​

On the DR site​

On the primary site​

Recover OCR tasks​

Start recovery​

API messages​

Recovery process​

Sync S3 content from primary to DR site​