9.1

Edit

How to switch between PROD and DR lines

Prerequisites

  • /home/taswf_prd/workfusion: <INSTALL_DIR> is the directory where WorkFusion software has been installed.
  • DB_ws_prod: WorkSpace and Sandbox PostgreSQL DB server from the PROD environment.
  • DB_ws_dr: WorkSpace and Sandbox PostgreSQL DB server from the DR environment.

Manage WorkFusion services on servers

To manage services that are running on a server, the wfmanager command is used.

  1. To manage a single service, use the following commands:

    $ source /home/taswf_prd/workfusion/environment.sh
$ wfmanager start service
$ wfmanager stop service
$ wfmanager restart service

  2. To manage all services at once, use the following commands:

    source /home/taswf_prd/workfusion/environment.sh
$ wfmanager status all
$ wfmanager stop all
$ wfmanager restart all
$ wfmanager start all

  3. When you run wfmanager to start all commands, the services are started in strict order, which is controlled by the Inorder service. There is no need to manage the starting or stopping of services in a strict order by yourself.

    Known behavior: "inorder FATAL Exited too quickly (process log may have details)" if other services are in RUNNING state, it means Inorder service done his job correctly.

Start and stop Tableau service on BI host

  1. Use the following commands to manage the Tableau service on the BI host:

    cmd
> cd D:\WorkFusionServer\10.5\bin
> tabadmin start
> tabadmin stop
> tabadmin restart

  2. Alternatively, in Windows Tray, find the Tableau Server icon, right-click the icon, and choose an action to be executed.

Stop services on each line of PROD or DR environments

If a component, except the APM server or services, goes down on the PROD or DR line, and there is a need to switch this line off and switch it to the DR or PROD line, do the following:

  1. Actions applied to the PROD or DR line servers must be taken in the following order:

    ML slaves → ML Master → APM → APP → DB

  2. To stop all services on each server of a line, run the following:

    source /home/taswf_prd/workfusion/environment.sh
$ wfmanager stop all

  3. Switch the DNS name for the line of PROD/DR env to the DR/PROD line.

Start services on each line of PROD or DR environments

Actions applied to PROD or DR line servers, except the WorkSpace APP and DB servers, must be taken in the following order: DB → APP → APM → ML Master → Ml slaves.

To start all services on each server of a line, run the following:

source /home/taswf_prd/workfusion>/environment.sh
$ wfmanager start all

Stop and start services on WorkSpace APP and DB servers for each environment

To keep replication going, keep the PostgreSQL service running all the time on the DB_ws_prod and DB_ws_dr servers.

  • Actions to stop the WorkSpace and Sandbox services must be taken in the following order: APP_workspace_server → DB_workspace_server.

  • Actions to start the WorkSpace and Sandbox services must be taken in the following order: DB_workspace_server → APP_workspace_server.

Switch to other servers

Scenario 1

The APP_workspace server on the PROD environment goes down, or some services on the APP_workspace server go down, and there is a need to switch APP_workspace and DB_workspace servers of the PROD environment to the same servers of the DR environment.

To do that, follow the steps below:

  1. Stop all services on the APP_workspace server of the PROD environment. This step is only valid if a service on the APP_workspace server goes down.
  2. On the DB_workspace server of the PROD environment, stop all services, except PostgreSQL.
  3. Switch DNS names to point to APP_workspace and DB_workspace of the DR environment.
  4. On the DB_workspace server of the DR environment, start the services. It is assumed that the PostgreSQL service is already running to keep replication going.
  5. Configure the replication from the DB_workspace DR server to the DB_workspace PROD server. For that, change the PostgreSQL node roles.
  6. On the APP_workspace server of the DR environment, start all services.

Scenario 2

The DB_workspace_server on the PROD environment goes down, and there is a need to switch the APP_workspace and DB_workspace servers of the PROD environment to the same servers of the DR environment.

To do that, follow the steps below:

  1. On the APP_workspace server of the PROD env, stop all services.
  2. Switch DNS names to point to APP_workspace and DB_workspace of DR env.
  3. On the DB_workspace server of the DR env, start the services. It is assumed that the PostgreSQL service is already running to keep replication going.
  4. On the APP_workspace server of the DR env, start all services.
  5. When the DB_workspace server of the PROD env is repaired and the PostgreSQL service is running, configure the replication from the DB_workspace DR server to the DB_workspace PROD one.

The best case is to keep the PostgreSQL service running all the time on the DB_workspace servers of PROD and DR environments to keep replication going.

If there is a need to switch the DNS names for the APP_workspace and DB_workspace servers of any environment, leave the PostgreSQL service running (except for the case when a DB server goes down) to keep replication going.

Scenario 3

The APP server in the PROD or DR environment goes down, or some services on the APP server go down, and you need to switch the whole line from the PROD or DR environment to the same line of the DR or PROD environment.

To do that, follow the steps below:

  1. Stop all services on all Linux servers of that line in the following order: ML slaves → ML Master → APM → APP → DB.
  2. Switch DNS names to point to the same line on the DR or PROD environment.
  3. Start the services on all Linux servers of the line in the following order: DB → APP → APM → ML Master → Ml slaves.
  4. Restart the Tableau Service on the BI host of the DR or PROD environment.
note

After switching any line from PROD to DR to DR or PROD accordingly, it is required to restart the Tableau service on BI host of the DR or PROD environment.

Switch replication between WorkSpace PostgreSQL DB servers in PROD and DR environments

The replication between the DB_ws_prod and DB_ws_dr servers is configured using the IP addresses of those servers, not DNS names. This is done to get round the issue related to switching DNS names. The DB_ws_prod server is a Master server (Master PostgreSQL node), and the DB_ws_dr server is a slave server (a standby PostgreSQL node).

If there is a need to switch the replication from a Master node to a slave node or vice versa, run the following command on the slave node:

/home/taswf_prd/workfusion/postgres/bin/repmgr -f /home/taswf_prd/workfusion/postgres/repmgr/repmgr.conf standby switchover

The output looks as follows:

NOTICE: executing switchover on node "10.213.64.228" (ID: 2)
NOTICE: local node "10.213.64.228" (ID: 2) will be promoted to primary; current primary "10.212.57.204" (ID: 1) will be demoted to standby
NOTICE: stopping current primary node "10.212.57.204" (ID: 1)
NOTICE: issuing CHECKPOINT
DETAIL: executing server command "source /home/taswf_prd/workfusion/environment.sh && wfmanager stop postgresql"
INFO: checking for primary shutdown; 1 of 60 attempts ("shutdown_check_timeout")
NOTICE: current primary has been cleanly shut down at location 0/B1000028
NOTICE: promoting standby to primary
DETAIL: promoting server "10.213.64.228" (ID: 2) using "pg_ctl  -w -D '/home/taswf_prd/workfusion/postgres/data' promote"
DETAIL: waiting up to 60 seconds (parameter "promote_check_timeout") for promotion to complete
server promoting
NOTICE: STANDBY PROMOTE successful
DETAIL: server "10.213.64.228" (ID: 2) was successfully promoted to primary
NOTICE: setting node 1's primary to node 2
NOTICE: starting server using "source /home/taswf_prd/workfusion/environment.sh && wfmanager start postgresql"
NOTICE: replication slot "repmgr_slot_2" deleted on node 1
NOTICE: NODE REJOIN successful
DETAIL: node 1 is now attached to node 2
NOTICE: switchover was successful
DETAIL: node "10.213.64.228" is now primary and node "10.212.57.204" is attached as standby
NOTICE: STANDBY SWITCHOVER has completed successfully
bash-4.2$

Check the cluster status by running the following command:

/home/taswf_prd/workfusion/postgres/bin/repmgr -f /home/taswf_prd/workfusion/postgres/repmgr/repmgr.conf cluster show

The output looks as follows:

ID | Name    | Role    | Status    | Upstream      | Location | Connection string
---+---------+---------+-----------+---------------+----------+-------------------
 1  | 10.212.57.204 | standby |   running | 10.213.64.228 | default  | host=10.212.57.204 port=5432 user=repmgr dbname=repmgr
 2  | 10.213.64.228 | primary | * running |               | default  | host=10.213.64.228 port=5432 user=repmgr dbname=repmgr
← PreviousNext →