Edit

General configuration

Agent running

Prerequisites: JRE 8

To run the agent, use the below code. Change <VERSION> to an appropriate value:

java -Xmx512m -jar status-agent-<VERSION>.jar --spring.config.location=classpath:/application.yml ## With configuration file inside .jar
java -Xmx512m -jar status-agent-<VERSION>.jar --spring.config.location=/opt/status/application.yml ## With configuration file placed on the server

Logs

Logs are created relatively to the application home directory in logs.

The absolute path to the log directory can be specified via -DLOG_DIR=.

Configuration

Below is the default configuration (no need to specify in a custom config file):

info:
  company: WorkFusion
  build.version: "@project.version@"
  build.timestamp: "@timestamp@"
  
pools:
  executor:
     queuecapacity: 100
     corepoolsize: 2
     maxpoolsize: 100
      
reporter:
  console:
    enable: true
    interval: 1
  
credentials: []
groups: []
alerts: []
components: []
checks: []

Simple custom configuration example:

groups:
  - { id: vds, name: "VDS Services"}

components:
  - id: gateway
    name: VDS Gateway
    group: vds
    description: "Version %gateway.info.version%"
    alerts: [ gateway ]
 
alerts:
  - id: gateway
    triggers:
      - { type: performance,  metric: gateway.info.time, gte: 10000 }
      - { type: major_outage, metric: gateway.info.http.code, neq: 200 }   
 
checks:
  - id: gateway.info
    type: http
    expression: "http://localhost:8080/project-info"
    evaluate:
      - { id: http.json, arg: version, type: string, metric_name: version}
 

General settings

Concurrency settings

pools:
  executor:
     maxpoolsize: Number of threads in primary executor pool
     queuecapacity: Number of tasks in queue for thread pool (grows when pool is saturated)
     corepoolsize: Default number of threads

Unirest settings used for HTTP client configuration:

unirest:
  concurrency:
    maxTotal: 200
    maxPerRoute: 20
  timeouts:
    connection: 10000
    socket: 60000

Reporter settings

reporter:
  console:
    enabled: true ## Whether print all metrics to console
    interval: 1   ## Interval to print metrics (in minutes)
  slf4j:
    enabled: true ## Whether log all metrics using slf4j logger
    interval: 1   ## Interval to print metrics (in minutes)
  mongo:
    enabled: true ## Whether dump all metrics to MongoDB (required for metrics history and proactive checks)
    source: Mongo ## Datasource name of MongoDB database
    interval: 1   ## Interval to dump ametrics (in minutes)

Model back end

model:
  backend: memory ## Supported backend types: memory/mongo/cachet. Memory backend type doesn't have any persistence
  source: Mongo   ## Datasource name of MongoDB database. Not required for "memory" and "cachet" backends

Metrics back end

metrics.history:
    source: Mongo            ## Datasource name of MongoDB database where all metrics are stored
    cleanup: "0 0 0 * * *"   ## Schedule of cleanup for metrics data (once per day)
    days: 40                 ## Remove metrics older than 40 days

Environment

This is a key-pair values that can be used as values for some attributes later in a config (checks, data sources).

Environment keys can be referenced later via key:

environment:
  key1: value1
  key2: value2

Examples:

environment:
  ocr_hostname: localhost
  ocr_protocol: http
  ocr_port: 8080
 
checks:
  - id: ocr
    type: http
    expression: "%ocr_protocol%://%ocr_hostname%:%ocr_port%/api/project-info"

Credentials

User-password pairs that can be used within checks:

credentials:
  - id: Id of credentials pair
    user: user name
    password: password

Example for RabbitMQ:

credentials:
  - id: rabbitmq
    user: admin
    password: password
 
checks:
  - id: queue
    credentials: rabbitmq
    expression: "http://localhost:15672/api/overview"

Data sources

Data sources can be linked within a <Check> so that corresponding check types can retrieve data.

datasources:
  - id: ID of datasource
    type: One of supported drivers
    protocol: Protocol of datasource (http, https)
    host: Hostname or IP
    port: Port of resource
    db: Dabase name
    url: URL for connection
    user: User name
    password: Password

Supported data source types:

• mongo
• sql
• mysql
• postgresql
• elasticsearch
• http
• tcp

Examples:

• mongo data source

  datasources:
    - id: LocalMongo
      type: mongo
      host: localhost
      db: ocr
  

• mysql data source

  - id: LocalMysql
    type: mysql
    url: "jdbc:mysql://localhost:3306/test?characterEncoding=UTF-8&autoReconnect=true&useSSL=false"
    user: root
    password: root
  

• postgres data source

  - id: LocalMysql
    type: mysql
    url: "jdbc:postgresql://localhost:5432/postgres"
    user: root
    password: root
  

• Http data source

  - id: OcrRest
    type: http
    protocol: http
    host: localhost
    port: 8080
  

Groups

List of component groups:

groups:
  - id: Id of group (internal use, can be referenced from components)
    name: Name of group
    collapsed: Collapse the group? 0 = No. 1 = Yes. 2 = If a component is not Operational. Default is 1

Example:

groups:
  - { id: vds.services, name: "VDS Services"}

Components

List of components:

components:
  - id: Id of component (internal, can be referenced later)
    name: Name of component
    group: Id of group
    description: Description of component, metric values can be referenced with %metric_id%
    link: URL link for component
    enabled: true/false
    alerts: List of alert ids to react on

Example:

components:
  - id: vds.gateway
    name: VDS Gateway
    group: vds.services
    description: "Version %vds.gateway.info.http.json.version%"
    alerts: [vds.gateway]

Alerts

An alert is a scheduled validation of metrics using some evaluation functions.

alerts:
  - id: Id of alert
    description: Description of alert
    dependson: Metric id which should have valid status (used to check underlying status agent proxy). Higher preference than alert check attribute
    enabled: true/false
    templates: List of template objects which replace specific check properties
    triggers:
      - metric: Name of metric to check
        gt: Check that value is greater than
        lt: Check that value is less than
        gte: Check that value is greater than or equal
        lte: Check that value is less than or equal
        eq: Check that value is equal
        neq: Check that value is not equal
        message: Message for this particular metric check
        points: How many points (times) should match condition before trigger (Default is 3 which means that only 3rd failed check will trigger alert). 
        dependson: Metric id which should have valid status (used to check underlying status agent proxy).

Examples:

alerts:
  - id: disk.prediction
    checks: 
      - { metric: server.disk.forecast.5m, lte: 0, type: performance, message: "Disk will be used in 5 minutes" }
      - { metric: server.disk.forecast.1h, lte: 0, type: performance, message: "Disk will be used in 1 hour" }
      - { metric: server.disk.forecast.3d, lte: 0, type: performance, message: "Disk will be used in 3 days" }

Checks

Checks is a scheduled action producing metrics (based on supplied evaluation functions).

checks:
  - id: Id of check
    description: Human-readable description of check purpose
    type: One of supported check types
    file: Use file contents as expression result
    expression: Check type will use this to evaluate metrics
    schedule: Cron-like expression for periodic execution or number of seconds between checks
    tag: Tag which can be used for metric aggregation via "tag.*" functions
    evaluate: List of evaluation functions to get metric values
    templates: List of template objects which replace specific check properties
    enabled: true/false
    dependson: Other check id. Only run this check if dependson check is successful
    parameters: Additional parameters for specific checks (like analysis)
    sslFactory: true/false Use or don't use ssl factory for http-check
    ns: Namespace for evaluated metrics
    readTimeout: check request read timeout in milliseconds, default is 60000 (1 min)

Check types

• http
• http-json
• tcp
• command

The сommand check type can work in 2 modes:

• Default: run the command internally. This mode doesn't support pipes or redirection.
• Dump the expression content into a temporary file and execute it.

Example for Windows, default mode:

- id: command
  type: command
  expression: "curl -Il -s https://www.workfusion.com/"

Example for Windows, with temporary file:

- id: command
  type: command
  expression: "curl -Il -s https://www.workfusion.com/"
  parameters:
    executable: "cmd.exe /c"

Example for Linux, with temporary file:

- id: command
  type: command
  expression: "curl -Il -s https://www.workfusion.com/"
  parameters:
    executable: bash

• mongo
• sql
• java
• analysis

Evaluation functions

These functions process check results and extract additional metrics:

evaluate:
  - id: Id of function
    arg: Argument to function
    file: Use file contents as expression
    type: Object type of metric (default - "long")
    metric: override metric suffix name (default - name of function) (check id and function id are prepended)
    metric_name: override metric name (only check id is prepended)

How metric names are generated

Pattern: <CHECK_ID> + <METRIC_NAME>

Examples:

Metric: vds.gateway.resp

check:
  - id: vds.gateway
    evaluate:
      - id: http.code
        metric_name: resp

Additional patterns when metric_name is not specified:

<CHECK_ID> + <FUNCTION_ID>

Metric: vds.gateway.http.code

check:
  - id: vds.gateway
    evaluate:
      - id: http.code

<CHECK_ID> + <FUNCTION_ID> + <METRIC>

Metric: vds.gateway.http.code.resp

check:
  - id: vds.gateway
    evaluate:
      - id: http.code
        metric: resp

Schedule format:

• Cron expression: *20 \* \* \* \* \*. Run every 20 seconds.
• Fixed delay in seconds: 20 (default).

Examples:

• command check

- id: vds.gateway.info
  type: command
  expression: "curl -s http://localhost:8080/project-info"
  schedule: "20"
  evaluate:
    - { id: command.exit }
    - { id: command.output, type: string }

• mongo check

- id: ocr.db
  type: mongo
  expression: "{ dbStats: 1 }"
  schedule: "20"
  source: LocalMongo
  evaluate:
    - { id: mongo.object, arg: dataSize, type: long, metric: size}

• mysql check

- id: mysql.db.status
  type: sql
  expression: "show status like 'Threads_connected'"
  schedule: "20"
  source: LocalMysql
  evaluate:
    - { id: sql.output, arg: Value, type: long, metric: connections }

Evaluation functions list

Type	ID	Description	Arg	Generated metrics
http	http.json	Get a response body as JSON and extract an object.	1) Exact a key from a JSON object. Examples: key, key.child 2) register_all: registers all child key-value pairs to metrics. Used to retrieve stats from proxy Status Agent.	Default pattern
http	http.response	Get a response body as a value.		Default pattern
http	rabbitmq.summary	Get summary from the RabbitMQ stats.		messages, messages_ready, messages_unacknowledged
command	command.exit	Exit code of a command.		Default pattern
command	command.output	Output of the command.	errors: only add output to metric when the command is not successful.	Default pattern
command	command.cpu.util	Command to get the CPU utilization info. Provides a value of load from 0.0 to 1.0, taking into account available cores.		Load
command	table.value.index	With expression, for instance, cat /proc/meminfo, you can parse an output specifying needed args.	0,MemTotal,1: only returns the MemTotal value from the table output.	memtotal
mongo	mongo.object	Get an object from the mongo query result.	Exact a key to retrieve. Examples: key.	Default pattern
sql	sql.output	Get output from an SQL query.	Value of the specified column. Note: Arg can be omitted if the result returns a single value.	Default pattern
java	disk.space	Get the status from a disk.	Path to be checked for the disk space information.	total, free
http	rpa.manager.hubs	Parses the SG hub JSON.		running, max
http	rpa.manager.quotas	Parses the SG quota JSON.		running, max
java	expression	Calculates a new metric based on the expression. Template variables can be used. Metric names for templating are taken from the results of the previous evaluation functions.	Expression examples: 1) 1 + 1 2) %metric1% + %metric2%	Default pattern
java	metric	Fetches existing metric from registry into result of current check. Can be useful by pairing with subsequent "expression" function	1) rpa.hubs.all.missing 2) rpa.hubs.%hub1_hostname%.missing	Default pattern
java	tag.sum	Aggregation function to iterate all existing checks with a specific tag and sum their results. metric is the metric to aggregate. metric_name is the resulting metric name.	rpa.hub is the tag name.
analysis	anomaly.egads	Find anomalies in timeseries data.		Default pattern
analysis	forecast.regression	Forecast a metric value in the future.	Number of hours in the future	Default pattern
analysis	trend	Calculate a metric trend.		Default pattern

Processes

process config: configuration of managed processes with APM.

processes:
  - id: id of managed process
    expression: expression to start process
    stop: expression to stop process
    autorestart: flag whether to restart process in case of exit or exception
    startup: flag whether to start process after agent start
    tag: additional tag to group processes
    directory: working directory to evaluate start/stop expression

Dashboards

Dashboard is a collection of metrics to be displayed on a single page.

dashboards:
  - id: Id of dashboard
    metrics: List of metric names

Example:

dashboards:
  - id: workfusion
    metrics:
      - memory.utilization
      - cpu.spike
      - disk.decreasing

Template engine

The template engine allows creating multiple configuration objects using a single definition.

Sample check with 2 templates:

- type: http
  source: api
  templates:
    - { id: check1, expression: "/api/first" }
    - { id: check2, expression: "/api/second" }

The example above will be parsed, and 2 checks will be created similar to the following definitions:

- id: check1
  type: http
  source: api
  expression: "/api/first"
 
- id: check2
  type: http
  source: api
  expression: "/api/second"

Not all configuration objects and properties are supporting templates. Below is the comprehensive list of what is currently supported:

Configurations with templating support

Check

• id
• description
• expression
• source
• schedule
• file
• ns
• sslFactory
• credentials

Alert

• id
• description
• dependson
• schedule

Command

• id
• type
• schedule
• expression
• source
• credentials
• ns

Substituting variables

For all supported template properties, it is possible to use environment variables that can be replaced at the final phase of loading the configuration.

Example:

- type: http
  source: api
  templates:
    - { id: check1.%var1%, expression: "/api/first/%var1%" }
    - { id: check2.%var2%, expression: "/api/second/%var2%" }