Enable RPA unit monitoring in ELK

By default, ELK (Elasticsearch, Logstash, Kibana) has no checks for the RPA units. The statistics are available only in Bot Manager.

To add Heartbeat metrics for the RPA units manually, follow the steps below:

1. Download Heartbeat from the ELK website based on the OS type (x32, x64) and unpack it on the RPA server to RPA_DIR.

2. Copy $RPA_DIR\metricbeat\ssl to the heartbeat folder.

   

3. Replace $RPA_DIR\heartbeat\heartbeat.yml with heartbeat.yml. To download the file, click the link.

4. Open ta Bot Agent config file according to the used RPA topology:

   • RDP-in-RDP: $RPA_DIR\bot-agent\conf\bot-agent-master.yml
   • VDI: $RPA_DIR\bot-agent\conf\bot-agent-nordp.yml

   In the file, add the Heartbeat description to the following lines:

   • In Components:

     - id: rpa.heartbeat
     name: "Heartbeat processes"
     group: rpa.server
     alerts: [ rpa.heartbeat ] # no need in alert definition, just register component with fake alert id

   • In Alerts:

     - id: rpa.heartbeat
       triggers:
       - { metric: rpa.heartbeat.status, neq: 0, message: "Heartbeat process check command failed" }

   • In Processes:

     - id: rpa.heartbeat
       expression: "cmd /c start /wait heartbeat-pipeline.bat ${environment.agent.hostname} ${logstash.heartbeat-endpoints}"
       directory: "../heartbeat"
       tag: "heartbeat"

   • In Checks:

     - id: rpa.heartbeat
         description: "Find heartbeat process"
         type: command
         expression: cmd /c "tasklist | findstr heartbeat"
         evaluate:
           - { id: command.exit }

5. On the RPA server, create the $RPA_DIR/heartbeat/heartbeat-pipeline.bat file with the following content:

   @echo OFF
   FOR /f "tokens=1,2 delims= " %%a IN ("%*") DO (
   SET HOSTS=%%b
   )
   IF DEFINED PROGRAMFILES(X86) (GOTO 64BIT) ELSE (GOTO 32BIT)
   
   :64BIT
   @echo ON
   heartbeat.exe -c "heartbeat.yml" -E "name=%1" -E "output.logstash.hosts=[%HOSTS%]"
   GOTO END
   
   :32BIT
   @echo ON
   heartbeat.exe -c "heartbeat.yml" -E "name=%1" -E "output.logstash.hosts=[%HOSTS%]"
   GOTO END
   
   :END

6. Create the $RPA_DIR\heartbeat\monitors.d\unit1.yml file for the first RPA unit.

   Specify the host and name as the actual RPA server's hostname and set the port for URLs to worker_port from RPA_DIR\bot-agent\conf\units\unit1.yml:

   - type: http
     urls:
     - "http://localhost:15201/health"
     schedule: '@every 60s'
     timeout: 30s
     name: 'rpa_server_hostname.unit1'
     check.request:
       method: GET
       headers:
         'User-Agent': 'Mozilla'
         Host: rpa_server_hostname
     check.response:
       status: 200
     fields:
       service_group_name: rpa

   Create proper unitNUMBER.yml files for all units.

   For the VDI configuration, use RPA_DIR\bot-agent\conf\units\unit_nordp.yml to acquire the worker_port value.

7. On the MASTER server, in ZooKeeper, add the logstash.heartbeat-endpoints parameter:

   INSTALL_DIR/zookeeper/bin/zkCli.sh create /config/bot-agent/logstash.heartbeat-endpoints MASTER_DNS_NAME:4571

   For the High Availability installation, specify a list of Master servers in the logstash.heartbeat-endpoints parameter (in double-quotes, comma-separated):

   INSTALL_DIR/zookeeper/bin/zkCli.sh create /config/bot-agent/logstash.heartbeat-endpoints "MASTER1_DNS_NAME:4571","MASTER_DNS_NAME:4571","MASTER_DNS_NAME:4571"

8. Stop and start the WFSvc service (note that you will be logged off) or restart the RPA server to restart Bot Agent services.

   After the restart, the Heartbeat monitoring status is available in Kibana:

   

9. Optionally, to set up the email notification, use the example rpa_watcher.json:

   See example

   {
   "trigger": {
       "schedule": {
       "interval": "5m"
         }
   },
   "input": {
       "search": {
       "request": {
           "search_type": "query_then_fetch",
           "indices": [
           "heartbeat*"
           ],
           "rest_total_hits_as_int": true,
           "body": {
           "size": 25,
           "_source": {
               "includes": [
               "@timestamp",
               "monitor.name",
               "monitor.status",
               "agent.name"
               ]
           },
           "query": {
               "bool": {
               "filter": [
                   {
                   "range": {
                       "@timestamp": {
                       "gte": "now-1m"
                       }
                   }
                   },
                   {
                   "terms": {
                       "monitor.name": [
                       "rpa_server_hostname.unit1",
                       "rpa_server_hostname.unit2",
                       "rpa_server_hostname.unit3",
                       "rpa_server_hostname.unit4",
                       "rpa_server_hostname.unit5"
                       ]
                   }
                   },
                   {
                   "term": {
                       "monitor.status": "down"
                   }
                   }
               ]
               }
           }
           }
       }
       }
   },
   "condition": {
       "compare": {
       "ctx.payload.hits.total": {
           "gte": 1
       }
       }
   },
   "actions": {
       "email_action": {
       "throttle_period_in_millis": 900000,
       "email": {
           "profile": "standard",
           "priority": "high",
           "to": [
           "example@workfusion.com"
           ],
           "subject": "Health check error for [{{#ctx.payload.hits.hits}}{{_source.agent.name}} {{/ctx.payload.hits.hits}}]",
           "body": {
           "text": "Some services have been reported down times. \n \n DETAILS: \n {{#ctx.payload.hits.hits}} {{_source.monitor.name}} on host '{{_source.agent.name}}' at '{{_source.@timestamp}}' is down. \n {{/ctx.payload.hits.hits}}"
           }
       }
       }
   }
   }