HA environment monitoring
A dedicated HA Overview dashboard was created to facilitate the detection of inconsistencies in a high-availability (HA) environment.
It highlights the following information:
- Service statuses
- Nodes on which active service instances are running
Service statuses
In an HA environment, services can run in the active-active mode or active-standby mode. The following visualization displays the service state in the time-series format.
Each bar displays the number of nodes that reported a service as up and the number of nodes that reported the service as down. Data is aggregated at two-minute intervals.
For active-active services, the bar is green, in which case all nodes report services as up. For active-standby services, the bar is only partially green: exactly one node reports services as up, others are down.
As the time-series visualization is used, it is easy to spot service state changes over a period in the past. For example, when a service restarts, there is a "down" and "up" message from the same node at one time interval (two minutes). On the chart, this event looks like a spike in comparison to adjacent time intervals.
Active nodes
This chart is useful to spot which node is currently active for a particular service to further conduct a more in-depth investigation. You only have to hover over the service of interest.
Failover events
This chart is useful for understanding whether the failover script caused a service to restart. In a perfect world, when all services are just running as expected, the chart should be empty.
When the failover script performs an action with a service, the action is displayed on the chart, together with the name of the node and service to which it is related:
In addition to this chart, the dashboard has the Saved search element. It allows you to grasp several events instead of selecting them one by one on a chart, for example, when multiple events happen in a close time interval.
Sample failover messages
| Message | Interpretation | Is it harmful |
|---|---|---|
| Stop process | The service was running but was forced to stop by the failover daemon because the node is not the leader anymore for that service. | Yes |
| Start process | The service was started by the failover daemon because the node became the leader for that service. | Yes |
| Process status is correct. No actions taken | Normal messages reported by each daemon every %timeinterval% seconds from all nodes (leader and followers) | No |
| ERROR, Update Nexus indexes task was not scheduled | The attempt to promote a new node to the leader for the Nexus service failed. | Yes |
| Service is unstable | The service failed several health checks on the current node and will be failed over to another node. | Yes |
| Node role changed from | The node changed its role for a particular service—either was demoted to a follower or promoted to the leader. For details, see full messages. | Yes |