9.0

Edit

Warm StandBy

Summary

A warm backup site is already stocked with hardware representing a reasonable facsimile of that found in your data center, the connectivity is also already established.

Current solution provides the duplication of critical components or functions of a system with the intention of increasing reliability of the system. When critical solution components (hardware power supplies, CPUs, data storage devices) fail or become degraded, solution failures can be avoided or minimized by providing redundant copies of these components that can be brought on-line quickly or that operate in parallel to their counterparts.

The Business Continuity considers the following list of redundant resources, assuming that the servers technical specification is the same as defined in the installation guide.

Each component contains its own operational data, and overall system data integrity should be preserved across the layers.

System Alias Warm Site Notes
Database db.* Yes
ML ml.* Yes
OCR ocr.* Yes
PM pm.* Yes Provides less business critical functions. RPO and RTO can be relaxed for the solution cost reduction.
APP wfapp.*, wfapp-s3.*, (bucket).wfapp-s3.* Yes
BI bi.* Yes

Provides less business critical functions. RPO and RTO can be relaxed for the solution cost reduction.

Data integrity

The section describes how the solution will fully account for the methods for handling data integrity – such as queuing or real time backup.
Data integrity must be planned for to prevent data loss or corruption that may result in significant disruption in the solution, thus impacting the users and potentially the business.
There are following Data Integrity enforcement measures in the WorkFusion system DR architecture:

  • Regular DB backups provide a high-fidelity restoration points to the system;
  • Internal system mechanisms of exceptions handling allow to fail-over in case of minor data integrity loss without operation level degradation;

Business critical system components also have HA Solutions are based on a synchronous NAS replication which grants transnational integrity, allowing to minimize related risks. Other components are using data snapshots which also grant instance-based integrity.

System recovery

To restore service, the last backups from your off-site storage facility must be delivered, and bare metal restoration completed, before the real work of recovery can begin.

The use of a redundant system[s] operating in parallel with a primary system prevents the loss of a minimal amount of data and is used to reconstruct the data on the primary system. The Disaster Recovery section identifies and describes the procedures for keeping warm stand by systems in sync with the primary system and for starting up their operations, and the procedures for reconstructing lost or corrupted data.

DNS switch over

It is required to have DNS names for the applications and database servers, so that smooth switch over could be done in case of disaster recovery i.e. switch the applications/database from one server to another (warm site). Fail over from a main site to the Warm Site occurs by a global DNS switch approach as the most efficient software DR best practice.

The communication between servers should be configured by using DNS aliases. Installation procedure assumed DNS aliases applied instead of IP's or hostnames.

Required DNS aliases, passive DR zones application can have their own dns post-fixed with dr (Recommended). The recommended dr prefix is for testing the dr instances independently.

Server DNS (Active) Notes
Application (Workfusion App) wfapp.<customername>.com Workfusion apps including Control Tower, Workspace.
Database (MySQL, MongoDB, PostgresSQL) db.<customername>.com:<Port> Different DB services are identified using port., for example, MySQL Port is 3306
Machine Learning ml.<customername>.com
OCR ocr.<customername>.com
RPA rpa.<customername>.com The rpa servers are clustered machine (Windows)

Business Intelligence

 bi.<customername>.com
Performance Monitor apm.<customername>.com

Disaster recovery plan

It is important to ensure that the WorkFusion system Warm Site hosting time should be minimized in order to provide optimal performance and usability to business operations. This requirement can be achieved by focusing on Main Site recovery operations right after the Fail Over to the Warm Site operations completed.

To get the sequence of switching back from active Warm Site back to restored Main Site, please use the recommended sequence.

Sequence 1: Performance Monitor (APM/PM)

WorkFusion Performance Monitor is a proprietary system, allowing to measure and control the application health by analyzing multiple metrics of the target system. This application component should be restored before any other components. This is required to monitor the health of other application/system components switch over.

The DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Recovery procedure

Logi n to APM server via SSH and run the following commands:

INSTALL_DIR/apps/wfagent/wfagent start

http://apm.*:10101/components Note: update period of components state is about 3 minutes.

Sequence 2: Data base/S3 recovery (data integrity)

The data base and the S3 data store should be recovered once sequence 1 is done, this is an important step for forthcoming applications to be up and running.

Database

After validation that replicated DB datafiles are in consistent state, start the necessary RDBMS services (see details below) namely :

  1. MySQL
  2. PostgreSQL
  3. Solr & MongoDB

If DB datafiles are not in consistent state, DB backup restore operation might be required (see 4.2. section).

S3 Data Store

S3 Datastore is a No-SQL data storage system built on top of Riak platform to provide AWS-S3-compliant API. The Datastore is used by the core product itself as well as by client apps and Business Processes, which means that the data loss prevention measures should be applied for this component.

Riak-based systems allow eventual consistency of the data, so NAS replication is sufficient to provide reliable restoration point with minimal data loss.

Recovery procedure

  1. Log in to DB server via SSH and run the following commands:

    INSTALL_DIR/apps/wfagent/wfagent start
INSTALL_DIR/apps/s3emu/s3emu.init start
 
rm -f INSTALL_DIR/apps/mysql/subsys/*
rm -f INSTALL_DIR/apps/mysql/data/*.pid
INSTALL_DIR/apps/mysql/mysql.service start
chmod +x INSTALL_DIR/apps/pgsql/9.4/data/postgresql-9.4.service
INSTALL_DIR/apps/pgsql/9.4/data/postgresql-9.4.service start
INSTALL_DIR/apps/mongodb/mongod.service start
INSTALL_DIR/apps/solr/solr.init start

  2. Switch over to the Warm Site instance occurs in the same way, by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

  3. Open APM Web UI and check the current component state http://apm.*:10101/components.

Note: update period of components state is about 3 minutes

Sequence 3: Application and services

WorkFusion Core Application is a Java Tomcat system which uses DBs and S3 Document Storage for data management so the fail over of this component will depend on the results of underlying levels restoration.
Other app data is a temporary operational data which might not be preserved, and system log files which will be replicated by the NAS Mirror. The switch over to Warm Site instance occurs the same way, by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Recovery procedure

  1. Log in to APP server via SSH and run the following commands:

    INSTALL_DIR/apps/wfagent/wfagent start
INSTALL_DIR/apps/nginx/nginx.service start
INSTALL_DIR/apps/php-fpm/php-fpm.service start
INSTALL_DIR/apps/rpa_manager/rpa_manager start
INSTALL_DIR/apps/webapps/tomcat.init start
INSTALL_DIR/apps/nlp-uima-rest/tomcat-nlp-uima-rest.init start
INSTALL_DIR/apps/tabula/tabula start

  2. Open APM Web UI and check current components state http://apm.*:10101/components.

Note: update period of components state is about 3 minutes.

Sequence 4: OCR Server

OCR Services is a shared stateless component which only stores current in-progress request so the NAS Mirroring is sufficient to provide a data required to recover the service. In case of any in-progress entry is lost, the request might be resubmitted by the core application services. Fail over to the Local Server or the Warm Site instance occurs the same way, by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Recovery procedure

  1. Log in to OCR server via SSH and run the following commands:

    INSTALL_DIR/apps/wfagent/wfagent start
/var/lib/workfusion/apps/ABBYY/FREngine11/Bin/abbyy-fre11-licensing-service start

  2. Activate ABBYY OCR SDK as described in Activating OCR license offline.

  3. Run the command:

    INSTALL_DIR/apps/ocr/ocr2.init start

  4. Log in to APP server via SSH and reload nginx configuration:

    INSTALL_DIR/apps/nginx/nginx.service reload

Note: update period of components state is about 3 minutes.

Sequence 5: RPA

RPA cluster consists of a set of Windows-based machines with stateless components pre-provisioned. No operational data is required to be restored in order to recover the service, which means that the fail over procedure include only configuration switch.

Switch over to Warm Site instance occurs by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Sequence 6: Machine Learning (ML)

ML Services is a shared stateless component which operates with a two major types of data:

  • In-progress request data so the NAS Mirroring is sufficient to provide a data required to recover the service. In case of any in-progress entry is lost, the request might be resubmitted by the core application services.
  • Static rarely updated file-based data, which means that NAS Mirror can provide a reliable recovery of N-1 version of the model which is enough for the most cases. To ensure that the most recent data model is recovered, staging or backup data can be used.

Fail over to the Warm Site instance occurs the same way, by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Recovery procedure

  1. Log in to ML server via SSH and run the following commands:

    INSTALL_DIR/apps/wfagent/wfagent start
INSTALL_DIR/apps/rabbitmq/rabbitmq-server.service start
INSTALL_DIR/apps/ducc/ducc.init start
INSTALL_DIR/apps/vds/vds.init start

  2. Log in to APP server via SSH and reload nginx configuration :

    INSTALL_DIR/apps/nginx/nginx.service reload

Sequence 7: Business Intelligence (BI)

BI components is a Tableau-based service, used for system metrics collection, processing and visualization. This service is not required for a normal business operations of the WorkFusion platform, so the switch over procedure for this component allows of operational data lost between backup points (RPO is about 24 hours). Switch over from a main site to the Warm Site instance occurs by a DNS switch to preserve a component FQDN while an actual endpoint IP address might differ.

Recovery procedure

Diaster Recovery strategy for WorkFusion Analytics is to restore data from latest backup or a Virtual Machine snapshot.

Restore primary server to secondary server:

  • Copy backup's file from primary (or NAS) to DR instance
  • Open command line and run commands:
cd <INSTALL_DIR>\9.3\bin
tabadmin stop
tabadmin restore <backup_path_and_filename>
tabadmin start

Installation of WorkFusion for DR

When installation WorkFusion components for DR - there are general and component-specific principles. This guide will describe general setup of components for DR and then, where applicable, component-specific details

Use the latest installation guide for details on installation steps and prerequisites which will be mentioned below.

Primary datacenter (Main Site)

Prerequisites:

  • Request DNS aliases assigned to all instances
  • Shared volume (NAS) is attached to some location on all Linux machines

Installation steps:

  1. Go through the Infrastructure preparation section of the installation guide.
  2. Set up all components using shared directory as installation path (INSTALL_DIR).

DR datacenter (Warm Site)

Installation steps:

Go through the Infrastructure preparation section of the installation guide.

APP installation

DR datacenter (Warm Site)

Additional installation steps:

Copy all crontab task for root and WF_APPS_USER from Primary datacenter to the Warm datacentre:

crontab -l
crontab -u WF_APPS_USER -l

DB installation

DR datacenter (Warm Site)

Additional installation steps:

Copy all crontab task for root and WF_APPS_USER from Primary datacenter to the Warm datacentre

crontab -l
crontab -u WF_APPS_USER -l

OCR installation

DR datacenter (Warm Site)

Additional installation steps:

  1. Install ABBYY SDK to <INSTALL_DIR>/apps/ABBYY.

  2. Change a permission for the ABBYY folder.

  3. Activate OCR License. Use the previously activated license or a new one.

  4. Copy all crontab task for root and WF_APPS_USER from Primary datacenter to the Warm datacentre.

    crontab -l
crontab -u WF_APPS_USER -l

ML installation

DR datacenter (Warm Site)

Additional installation steps:

  1. Copy the WFUSER_HOME_DIR/.ssh and WFUSER_HOME_DIR/ducc from Primary datacenter to the same path in Warm datacentre.

  2. Make symlink.

    mkdir -p INSTALL_DIR/apps/ducc/eval
ln -s INSTALL_DIR/apps/ducc/eval WFUSER_HOME_DIR/ducc

  3. If ML instance uses custom SSH implementation, configure pass wordless authentication for WorkFusion user.

  4. Copy the WFUSER_HOME_DIR/.bash_profile file from Primary datacenter to the same path in Warm datacentre.

  5. Copy the /etc/ld.so.conf.d/workfusion.conf file from Primary datacenter to the same path in Warm datacentre.

RPA installation

DR datacenter (Warm Site)

Installation steps:

Set up RPA Server the same way as in Primary datacenter. For mode details, see the RPA installation guide.

BI installation

Primary datacenter

Installation steps:

  1. Follow the WorkFusion Analytics installation guide to install necessary software.

  2. Schedule periodic (daily) backups.

  3. Open the commandline and run commands:

    cd  <INSTALL_DIR>\9.3\bin
tabadmin backup <backupfilename> -d
copy <original_backup_path_and_filename> <network_drive_or_other_location_path_and_name>

DR datacenter (Warm Site)

Installation steps:

  1. Set up BI Server the same way as in Primary datacenter.
  2. Activate a license.
  3. Stop BI Server processes.
← PreviousNext →