Learn about failover script technical details
IA Cloud Enterprise includes several components that don't support high availability (HA) out-of-the-box. The components can't be executed simultaneously and handle client requests.
To implement HA for the above cases, you need an additional custom component acting as an arbiter. It runs in the background, collects the target component status, and ensures that at most one healthy component instance is up and running.
The custom component can be especially useful in the following cases:
- A reliable distributed lock mechanism is required to ensure that only one instance can be up and running.
- It is important to allow for complex failover scenarios, for example, the failover of two to three services in case of an issue in one service.
- A possibility is essential to implement HA for any component.
- It is not recommended to depend on packages requiring root permissions to be installed or available only in a special Red Hat subscription.
Design
The solution consists of two scripts, and each focuses on one aspect of the solution. ZooKeeper provides central state storage and enables the handling of locks or elections.
failover.pyis responsible for process management. It establishes a connection to ZooKeeper and gets the ID of the current ACTIVE node. If the node ID of the server on which the script is executed matches the ACTIVE node ID,failover.pystarts a target process using Supervisord.watchdog.pyis responsible for monitoringfailover.pyand process dependencies. It runs periodic checks and does the following:- Stops the target process if the corresponding
failover.pyscript isn't running. - Restarts
failover.pyif its monitoring endpoint fails to return the 200 status code.
- Stops the target process if the corresponding
Dependencies
Components depend on the following entities:
| Component | Feature | Use case |
|---|---|---|
| Python2 | Runtime and external libraries | - |
| Kazoo | ZooKeeper API |
|
| Flask | Web framework | Check data exposure via REST API |
| ZooKeeper | HA |
|
| Supervisord | Server process management |
|
Components depend on the following failover scripts:
| Component | Feature | Use case |
|---|---|---|
| Nexus | HA |
|
| WorkFusion | HA |
|
| WorkSpace | HA |
|
| SQC | HA |
|
| Bot-manager | HA |
|
| Task-dispatcher-service | HA |
|
| Automl-gateway-service | HA |
|
| Automl-model-service | HA |
|
| Automl-model-management-service | HA |
|
| Worker-management-service | HA |
|
failover.py
Script arguments
| Argument | Description | Default value | Example |
|---|---|---|---|
-c | Configuration file path | - | /tmp/configuration.ini |
Script API
| Endpoint | Request | Response | Purpose |
|---|---|---|---|
/api/v1/maintenance/enable | GET | String | Enable the script maintenance mode |
/api/v1/maintenance/disable | GET | String | Disable the script maintenance mode |
/api/v1/maintenance | GET | String | Maintenance mode status |
/api/v1/health | GET | String | Target service health check report |
/api/v1/status | GET | String | Target node status check report |
/api/v1/monitoring | GET | String | Script process running check |
watchdog.py
Script arguments
| Argument | Description | Default value | Example |
|---|---|---|---|
-c | Configuration file path | - | /tmp/configuration.ini |
Script API
| Endpoint | Request | Response | Purpose |
|---|---|---|---|
/api/v1/maintenance/enable | GET | String | Enable the script maintenance mode |
/api/v1/maintenance/disable | GET | String | Disable the script maintenance mode |
/api/v1/maintenance | GET | String | Maintenance mode status |
/api/v1/dependencies | GET | String | Report on node dependencies |
/api/v1/monitoring | GET | String | Script process running check |
Node status check
The failover.py script starts a separate process that periodically executes the node status check. This status check is an HTTP request to the watchdog monitoring endpoint. failover.py expects watchdog.py to return the 200 status code.
The latest result of the node status check is stored in ZooKeeper and script data structure and shared among all processes.
Service health check
The failover.py script starts a separate process that periodically executes a service health check.
- If a node is the leader, the health check is an HTTP request to the service monitoring endpoint.
failover.pyexpects the service to return the 200 status code. - If a node is a follower, the health check HTTP request isn't sent.
The latest result of the node status check is stored in ZooKeeper and script data structure and shared among all processes.
Leader election
The configuration example is as follows:
[application_process_leader_election]
execution_period = 10
execution_timeout = 10
service_start_counter = 10
cluster_unstable_counter = 15
leader_znode_lock_time_seconds = 50
The failover.py script starts a separate process that periodically executes the cluster leader election. Each iteration takes the execution_timeout + execution_period seconds.
A node could take part in the leader election only if it has passed the status check.
If a node wins a leader election, it obtains a lock on the leader znode for leader_znode_lock_time_seconds. During this time, no other node in the cluster can win the leader election and become the leader. Further, if the node participates in the leader election, it prolongs the znode lock time with each iteration. When the node is promoted from a follower to a leader, it remains the leader for service_start_counter iterations. During this time, the service must start on the leader node and pass a health check.
Before the leader election, with each iteration, the script checks the follower's status or checks the status and runs a health check of the leader. If the status check or health check on the leader node fails cluster_unstable_counter times in a row, the node is demoted to a follower and doesn't participate in the leader election for service_start_counter iterations. During this time, another node must be promoted to the leader and start the service.
Troubleshooting
You can experience issues with the leader election process. The most common ones are enumerated below.
First service start
- By default, a node starts as a follower.
- If the status check is passed, the node starts to participate in the leader election.
- If the node wins in the leader election, it locks the leader znode for 50 seconds and becomes the leader.
- The next
service_start_counteriterations, it prolongs the znode lock without the service status and health checks. It takes ~200 seconds. - After that, before prolonging the leader znode lock, status and health checks are executed. If the status check or health checks on the leader node fail
cluster_unstable_countertimes in a row, it is demoted to a follower and doesn't participate in the leader election forservice_start_counteriterations.
Process is unstable
- A node is the leader.
- If the status and health checks fail
cluster_unstable_countertimes in a row (~300 seconds), it is demoted to a follower and doesn't participate in the leader election forservice_start_counteriterations (200 seconds). During this time, another node must be promoted to the leader and start the service. - The node is a follower. If a status check is passed, the node starts to participate in the leader election.
failover.py is unstable
- A node is the leader.
- If
failover.pyraises an exception and doesn't participate in the leader election forleader_znode_lock_time_secondsseconds, the znode lock expires, and another node is promoted to the leader. It takes ~60 seconds.
Server is down
- A node is the leader.
- If
failover.pyis down and doesn't participate in the leader election forleader_znode_lock_time_secondsseconds, the znode lock expires, and another node is promoted to the leader. It takes ~60 seconds.
Iteration
tip
For more information, refer to the following pages: