Hardware

Hardware and OS

Work.AI requires a minimum of five servers to be prepared in the production environment. You can scale the environment to any extent by adding extra Agent and RPA servers if needed.

The product can be installed in two modes:

• Basic

  This mode requires fewer servers but doesn't provide high availability. See the Basic environment tab.

• High availability

  High availability (HA) is the ability of a system to be continuously operational for an extended period. Its aim is zero data loss and a maximum of 10 minutes of service recovery in case of failure, after which it must be fully restored. For that, you need 2-3 instances of each server that you want to be highly available. See the High-availability environment tab.

Basic

Server	OS (Recommended in bold)	CPU (cores)	RAM (GB)	Disk (GB)	ZK SSD (GB)	/vds-data (GB)
Master16	RHEL/CentOS 8.x or 7.9 Rocky Linux 8.9	8	32	750	20	250
Agent (1-3 servers)2,6	RHEL/CentOS 8.x or 7.9 Rocky Linux 8.9	8	32	150	-	-
Analytics3,4	Superset (default): RHEL/CentOS 8.x RHEL/CentOS 7 or earlier versions are not supported. Rocky Linux 8.9	2	4	50	-	-
MS SQL	Microsoft Windows Server 2022, 2019, 2016	4	16	500	-	-
RPA (5 bots)3	Enterprise Server: Microsoft Windows Server 2019, 2016, 2022 VDI Server: Microsoft Windows Server 2019, 2016, 2022 Microsoft Windows 10 Microsoft Windows 11	8, 2.8 GHz+	16	150	-	-
ML Ops Compute7	Kubernetes8	16	64	500, disk reserved for ML Ops only	-	-
ML Ops Training7	Kubernetes8	12	96 RAM, 40vRAM (Nvidia A100)	500, disk reserved for ML Ops only	-	-
ML Ops Inference7	Kubernetes8	16	32 RAM	500, disk reserved for ML Ops only	-	-

High Availability

Server	Number of servers	OS (Recommended in bold)	CPU (cores)	RAM (GB)	Disk (GB)	ZK SSD (GB)	/vds-data (GB)	/shared (GB)
Master16	3	RHEL/CentOS 8.x or 7.9 Rocky Linux 8.9	8	32	750	20	250	500
Agent6	3	RHEL/CentOS 8.x or 7.9 Rocky Linux 8.9	8	32	150	-	250	-
Superset Analytics (default)3	2	RHEL/CentOS 8.x (RHEL/CentOS 7 or earlier versions are not supported) Rocky Linux 8.x	2	4	50	-	-	-
MS SQL Server	2	Microsoft Windows Server 2022, 2019, 2016	4	16	500	-	-	-
RPA (5 bots)3	2	Enterprise Server: Microsoft Windows Server 2019, 2016, 2022 VDI Server: Microsoft Windows Server 2019, 2016, 2022 Microsoft Windows 10 Microsoft Windows 11	8, 2.8 GHz+	16	250	-	-	-
PROXY5	2	RHEL/CentOS 8.x or 7.9 Rocky Linux 8.x	4	16	50	-	-	-
ML Ops Compute7	At least 3	Kubernetes8	16	64	500 each, 1 disk per server	-	-	-
ML Ops Training7	Kubernetes8	12	96 RAM, 40vRAM (Nvidia A100)	500, disk reserved for ML Ops only	-	-

Note that MS SQL Server is not provided by WorkFusion and must be prepared and licensed independently.

The OCR server is implemented as a service running on Agent servers. Keep that in mind when deciding about the number of Agent servers.

¹—Make sure that at least 15% of disk space on the server is always available.

²—At least one AGENT server (8 CPU / 32 GB RAM) is required to successfully install Work.AI. This is enough for PoC, Demo, or development environments with low workloads. For production and development environments where you plan running models training a minimum of three AGENT servers must be provided. The overall Product capacity depends on the number of AGENT servers as they perform all work during Business Processes execution. Typically the number of AGENT servers for production setups varies from 3 to 12. You can scale up the amount of servers after the installation, if your workloads increase with time or you experience performance issues. Depending on your planned workloads, the number of required AGENT servers must be discussed and agreed with your Delivery Manager.

³—The servers are only required if you plan to use the capability. For Superset HA and disaster recovery setup, refer to the guide.

⁴—Only for the Azure internal load balancer in the high-availability environment.

⁵—Make sure that at least 16GB of disk space is available in the / partition for RAM swap.

⁶—Using burstable instances is not recommended in case of the AWS deployment.

⁷—The ML Ops platform is an optional Work.AI component. For detailed installation requirements and instructions, read Install ML Ops Platform.

⁸—You can use both a standalone Kubernetes installation and Kubernetes as a service. ML Ops supports the deployment to Kubernetes cloud environments, such as Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), and Amazon Elastic Kubernetes Service (Amazon EKS). The prerequisites are the ability to enable node pools that can scale to zero and node pools that can provide GPU capabilities for model training.

Disk space and file system

• A separate non-root partition must be prepared for Work.AI to be installed and run. It can be a separate disk mounted to the server or a partition on the same disk. This partition is used to store data for all services, except ZooKeeper's datalog.

• A separate SSD device with at least 20 GB of space must be prepared and mounted to store ZooKeeper's datalog entries.

• The installation is performed via Ansible, which requires at least 8 GB of free space for temporary files in the /tmp directory on Linux servers. After the installation or update, the components are allocated in the file system only within the installation directory and do not use space allocated for the operating system.

• The root disk with the operating system must have reasonably enough space to keep Red Hat family 7.x or 8.x functioning. For example, 20 GB is enough.

• For RHEL/CentOS servers, the XFS or EXT4 filesystems are required.

• The required minimum throughput of your machines for running the Product should be not less than 2,500-3,000 IOPS. For complex high-capacity operations, target for the higher values.

info

When mounting disks on the Linux server, remember to add corresponding entries to the /etc/fstab file, so the disks are mounted automatically at startup.

Installation directory

A separate INSTALL_DIR directory for the Work.AI installation package must be created on the Master server. The application user must be the owner of this directory.

To set up the installation directory (here, /opt/workfusion), on the Master server, run the following commands:

$ mkdir /opt/workfusion
$ chmod 0755 /opt/workfusion
$ chown wfuser:wfuser /opt/workfusion

ZooKeeper datalog directory

/dataLog is the dedicated directory where ZooKeeper stores the history of its transactions.

A separate disk must be prepared and mounted to a directory on all Master servers. The application user must be the owner of this directory.

To create the ZooKeeper /datalog directory:

1. Connect the SSD disks to each Master server (one disk per server).

2. On each Master server, create a mount point for the ZooKeeper datalog directory. It must be located at the path /zookeeper/datalog relative to INSTALL_DIR:

   $ mkdir -p /opt/workfusion/zookeeper/datalog
   $ chmod 0755 /opt/workfusion/zookeeper/datalog

3. Mount the connected SSD disk to the created directory. For more information, see the instruction.

4. Ensure the application user has the necessary filesystem permissions for the created directory:

   $ chown -R wfuser:wfuser /opt/workfusion/zookeeper

The monitoring of the separate disk is available in the Kibana's dashboard at https://app_hostname/kibana/app/dashboards#/view/Zookeeper-disk.

Shared directories

The following shared directories must be created:

• INSTALL_DIR/vds-data: used by BEP services.
• INSTALL_DIR/shared: for storing data for Nexus, S3, and Logstash. Only used in the high-availability mode.
• INSTALL_DIR/logs_aggregated. Optional. The directory is used to separate Logstash data from other services to decrease the load on the shared disk.

Supported types of the file system on shared disks:

• NFS
• GFS2
• GPFS
• AWS EFS

caution

Windows Server-based storage solutions (DFS, NFS, CIFS/SMB) are not supported.

When mounting a shared file system, select one of the following options depending on your installation mode:

Basic

Server	Directory on the server	Shared disk size
Master server	INSTALL_DIR/vds-data	250 GB
Agent servers	INSTALL_DIR/vds-data	-

The Master server is the storage host and shares its INSTALL_DIR/vds-data directory among Agents. Create the INSTALL_DIR/vds-data directory on the Master server and mount it to Agent servers.

Note: on the Master Server, the vds-data directory must be separate from the primary disk space. Thus, the total disk space on the Master server must be at least 1 TB:

• 750 GB for Work.AI
• 250 GB for vds-data

High Availability

The single directory must be mounted from the external NAS to the mount points on the Master and Agent servers.

Server	Mount point on the server	Shared disk size: Development or Pilot	Shared disk size: Production
Master Agent 1-3	INSTALL_DIR/vds-data	500 GB	250 GB
Master	INSTALL_DIR/shared	250 GB	500 GB

Note: on the Master Server, the vds-data and shared directories must be separate from the primary disk space. Thus, the total disk space on the Master server must be at least 1.5 TB:

• 750 GB for Work.AI
• 250 GB for vds-data
• 500 GB for shared

If you have the Superset server and use an external NFS server, you must mount the vds-data directory on the Superset server. See also Prepare for installation | Perform additional setup for Superset server.

Elasticsearch data directory

Optional

To improve the Work.AI stability and throughput under high load, the Elasticsearch data directory must be stored on a dedicated SSD disk with a minimum of 100 GB available free space:

To create Elasticsearch data directory:

1. Connect the SSD disk to the Master server.

2. On the Master server, create a mount point for Elasticsearch data directory. It must be located at /elasticsearch/data relative to INSTALL_DIR:

   $ mkdir -p /opt/workfusion/elasticsearch/data
   $ chmod 0755 /opt/workfusion/elasticsearch/data

3. Mount the connected SSD disk to the created data directory. For more information, see the instruction.

4. Ensure the application user has the necessary filesystem permissions for the created directory:

   $ chown -R wfuser:wfuser /opt/workfusion/elasticsearch

System updates

After you deploy the required Linux servers, follow the instruction:

1. Find out the exact version of the operating system by running the command on each Linux server:

   uname -a

   Specify the version in the checklist.

2. Make sure that you have the latest OS updates installed. For that, on all Linux servers, run the following command to see all package updates (if any):

   yum check-update

   If there are no available packages of newer versions, your system is up-to-date.

3. If the command displays any available OS updates, install them to the system:

   yum update -y

caution

If your system has outdated versions of OpenSSL and OpenSSH, you may encounter the following error during installation: "OpenSSL version mismatch. Built against 30000070, you have 30300020 [WARNING]: Unhandled error in Python interpreter discovery for host [your host]: Failed to connect to the host via ssh: OpenSSL version mismatch. Built against 30000070, you have 30300020."

To resolve this issue, update the OpenSSL and OpenSSH packages using the following command: yum update openssl openssh.

If the RHEL repository is unavailable, for example, because the server is not connected to the Internet, see the troubleshooting guide.

Other requirements

• For optimal work of the Work.AI components, use the latest Chrome version.

  • Work.AI also supports the Chromium-based Microsoft Edge browser.
  • Firefox is not fully supported and the correct Product work in it is not guaranteed.

• Make sure that the TLS 1.2 support is enabled in the browser.

• Enable JavaScript on the Analytics servers.

• Set the timezone to UTC on the Master and Agent servers. To verify or change your current timezone on the server, use the following commands:

  timedatectl status
  timedatectl set-timezone UTC

  Note that changing the system's timezone requires root user or sudo privileges.