Set up disaster recovery

Disaster recovery (DR) is the ability to continue services' operation in case of major outages, often with reduced capabilities or performance. Disaster recovery solutions typically involve manual activities. DR is aimed at minimal data loss, including up to 15 seconds for database loss, up to 15 minutes for file system data loss, and up to 4 hours of server downtime.

The DR architecture implies having two sites: primary and DR one. The primary set hosts the complete set of servers (either in the HA mode or without it) with running components. The DR site is the exact copy of the primary one, including servers' DNS and topology. All the servers on the DR site are running while the components on them are stopped.

The replication of data between the sites is performed every 30 minutes. If the primary site fails, the system is switched to the DR site with minimum data loss.

Requirements

The shared directory INSTALL_DIR/shared must be mounted to the MASTER servers both on the primary and the DR sites.
The shared directory INSTALL_DIR/vds-data must be mounted to the Agent servers both on the primary and the DR sites.
The shared NFS directory must be replicated between the primary and the DR sites.
MS SQL must support the Log Shipment and the Asynchronous Replication methods.
The client must configure the Global DNS server or Route 53 to enable site failover.
The DR site must mirror the following objects of the primary site:
- Servers' topology
- DNS records
- Credentials for services
- Keys and certificates in /opt/workfusion/ssl
- File /opt/workfusion/vault/keys/vault_keys.json
All the IA Cloud components must be stopped on the DR site.

Prepare DR

We don't limit you in ways of deploying the DR site. You can do it, as you have used to, while keeping in mind that the DR site must be the exact copy of the primary one, including the number of servers and the DNS names.

After setting the DR site, remember to disable the autostart of its processes. For that, on the DR site:

On all Linux servers, run the following command to stop the processes. The processes on other servers, including MS SQL, must be stopped manually.
```
wfmanager stop all
```
On each server, run the following command to rename the directory with the startup scripts:
```
mv INSTALL_DIR/supervisord/apps INSTALL_DIR/supervisord/apps_disabled
```

Configure DR failover

Switch from Primary site to DR

Failover is switching from a primary site to the DR one.

To switch to DR:

Stop all processes (CT, workfusion, etc.) on the primary site.

We recommend shutdown all servers with running processes for that. Alternatively, you can use the command wfmanager stop all to halt the processes on the Linux servers. The processes on other servers, including MS SQL, must be stopped manually. For example, to stop the Analytics service, on the BI server, run the following commands:
```
 ```bash
 $ tsm login
 $ tsm stop
 ```
```
Switch the DNS records and IP addresses from the primary site to the DR one.
Perform the failover procedure for MS SQL according to your standard process.
On the DR site, start all servers.
On each Linux server, run the following command to rename the directory with the startup scripts:
```
mv INSTALL_DIR/supervisord/apps_disabled INSTALL_DIR/supervisord/apps
```
On each DR MASTER server, start ZooKeeper:
```
wfmanager update zookeeper
```
On the DR site, restore ZooKeeper:
1. Log in to each MASTER server as WFUSER.
2. In the INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini configuration file, in the zookeeper_data_backup_archive parameter, specify the name of the latest data archive.
3. In the INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini configuration file, in the zookeeper_logs_backup_archive parameter, specify the name of the latest datalog archive.
4. On each MASTER instance, run the following command to start ZooKeeper:
```
python INSTALL_DIR/tools/zookeeper/bin/zookeeper-backup.py -c INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini -a restore
```
The backup files of ZooKeeper are stored in the directory, which is specified in INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini.
On the DR site, on any MASTER server, go to INSTALL_DIR/zookeeper/bin/, and run zkCli.sh to create the /disaster node.
```
./zkCli.sh -server ZK_CONNECTION_STRING create /disaster disaster
```
Here, ZK_CONNECTION_STRING is the address: port setting of all three ZooKeeper instances separated with a comma.
On the DR site, on each Linux server, start the processes:
```
wfmanager update
```
The processes on the Windows servers, including MS SQL, are to be started via the user interface or with specific initiation commands. For example, to run the Analytics service, on the BI server, run the following commands:
```
$ tsm login
$ tsm start
```

After the replication, Minio and Nexus are immediately up-to-date, as both these components are installed in a shared directory.

Switch from DR to primary site

Once an incident is resolved, you may want to switch from the DR site back to the primary one or to a new environment. For that:

On the DR site, on each MASTER server, restart the zookeeper-dr process to execute the ZooKeeper backup procedure:
1. Log in to each MASTER instance as WFUSER.
2. Run the command:
```
wfmanager restart zookeeper-dr
```
On the DR site, stop the processes on the Linux servers:
```
$ wfmanager stop marathon-apps
$ wfmanager stop all
```
The processes on other servers, including MS SQL, must be stopped manually. For example, to stop the Analytics service, on the BI server, run the following commands:
```
$ tsm login
$ tsm stop
```

Disable services' autostart by renaming directory with startup scripts:

mv INSTALL_DIR/supervisord/apps INSTALL_DIR/supervisord/apps_disabled

Optional. Stop all DR servers.
Switch the DNS records and IP addresses from the DR site to the primary one.
Perform the failover procedure for MS SQL.
On the primary site, replicate common_nfs_share from the DR site.
On the primary site, start all MASTER servers.

Stop all services and start only ZooKeeper:

$ wfmanager stop marathon-apps
$ wfmanager stop all
$ wfmanager start zookeeper

On the primary site, restore ZooKeeper:
1. Log in to each MASTER server as WFUSER.
2. In the INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini configuration file, in the zookeeper_data_backup_archive parameter, specify the name of the latest data archive.
3. In the INSTALL_DIR}/tools/zookeeper/conf/zookeeper-backup.ini configuration file, in the zookeeper_logs_backup_archive parameter, specify the name of the latest datalog archive.
4. On each MASTER instance, run the following command to start ZooKeeper:
```
python INSTALL_DIR/tools/zookeeper/bin/zookeeper-backup.py -c INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini -a restore
```
The backup files of ZooKeeper are stored in the directory, which is specified in INSTALL_DIR/tools/zookeeper/conf/zookeeper-backup.ini.
On the primary site, on any instance of the MASTER server, go to /INSTALL_DIR/zookeeper/bin/, and run zkCli.sh to create the /disaster node.
```
./zkCli.sh -server ZK_CONNECTION_STRING create /disaster disaster
```
Here, ZK_CONNECTION_STRING is the address: port setting of all three ZooKeeper instances separated with a comma.
On the primary site, start all other processes on MASTER servers.
```
wfmanager start all
```
Start other Linux and Windows servers.

The processes on the Windows servers, including MS SQL, are to be started via UI or with specific initiation commands. For example, to run the Analytics service, on the BI server, run the following commands:
```
$ tsm login
$ tsm start
```
Run a health check Business Process to verify that the system is restored correctly.

OCR tasks recovery

You can recover OCR tasks that might not have been processed in case of disaster on the entire platform.

You can start the recovery process on one OCR server (node) at a time.

Start recovery

The REST endpoint /api/v2/recovery was added to start recovery immediately and give a chance to tasks stuck in either IN_PROGRESS or QUEUED status being processed again.

There are two ways to initiate recovery:

Restart the OCR REST process manually. Recovery is started automatically once the app is up.
Send the POST request to API /api/v2/recovery to OCR endpoint. Recovery is started on one of the OCR servers in a cluster.

API messages

Message	Recovery state
Recovery is in progress	Recovery has started successfully
No tasks for recovery	Recovery process hasn't found any task for recovery
Recovery start was skipped because it had started by another service	User tries to start recovery when recovery is already in progress. Recovery request is skipped.
Recovery start is failed	Recovery cannot be started for some reason (for example, problems with connection to ZooKeeper)

Recovery process

recovery_status was introduced for a task as a database column to implement the recovery procedure. It is returned along with general task info in /getTaskStatus API.

Once the recovery process is started, OCR looks in the database to gather information about tasks with the InProgress, Queued, Submitted status.

Task state before recovery Task state after recovery

Submitted Submitted tasks cannot be recovered. OCR recovery marks their status as ProcessingFailed and sets RecoveryStatus to Skipped.

Queued and InProgress For Queued and InProgress tasks, the recovery process checks task data for consistency to ensure that recovery is meaningful for this particular task. Data consistency is checked between MSSQL and S3. If inconsistent task data (image files or pattern) is missing in S3 or MSSQL, a task is marked with the ProcessingFailed status, and the recovery status is set to Skipped.

If data is consistent between S3 and MSSQL, the recovery process sends the task to processing, so the task is pushed to RabbitMQ and waits its turn for processing. The recovery status is set to Success.

Task state before recovery	Task state after recovery
Submitted	Submitted tasks cannot be recovered. OCR recovery marks their status as `ProcessingFailed` and sets `RecoveryStatus` to `Skipped`.
Queued and InProgress	For Queued and InProgress tasks, the recovery process checks task data for consistency to ensure that recovery is meaningful for this particular task. Data consistency is checked between MSSQL and S3. If inconsistent task data (image files or pattern) is missing in S3 or MSSQL, a task is marked with the `ProcessingFailed` status, and the recovery status is set to `Skipped`. If data is consistent between S3 and MSSQL, the recovery process sends the task to processing, so the task is pushed to RabbitMQ and waits its turn for processing. The recovery status is set to `Success`.

WorkFusion Documentation

10.2

Hardware

Install Linux components

Keycloak administrator guide