Security advisories

warning

This document was updated on December 22, 2021.

If you have applied any fixes before that date, study the following guide carefully as it may contain critical information regarding the log4j resolution.

The IA Cloud Enterprise v10.1.7 and v10.2.4 releases include the log4j v2.17.0 update. If you have an immediate concern about this vulnerability, you can use the following remediation in the meantime.

Apply patch

To fix the vulnerability, you must apply the patch to all Linux and Windows servers.

note

The following instruction is a subject for frequent updates. For example, the link to the logpresso-log4j2-scan utility may be updated.

This guide uses /opt/workfusion as the installation directory. Adjust it to your environment in all places.

Prepare for patching

Before applying the patch, ensure that all BPs are stopped in the Control Tower interface.

Stop Business Processes

To stop all Business Processes, on the Master server, go to the /control-tower/execution/active-instances/ ZooKeeper node:

Log in to the instance with the installer.
In the INSTALL_DIR/zookeeper/bin directory, execute the ./zkCli.sh script to access the ZooKeeper's interface.
In ZooKeeper, execute the following command to view existing ZK nodes:
```
ls /control-tower/execution/active-instances
```
The command shows a similar output:

If you see an active instance in your result above, log in to Control Tower, go to the active Business Process, and clear any search filter.

If you see any Active Running BP, stop the BP before turning off all the components.

note

If you can’t stop the BP due to pending Manual Tasks in your current BP’s, follow the Pause Business Processes instruction.

Pause Business Processes

To pause a Business Process, do the following:

Log in to Control Tower, and pause all Business Processes from there.
On the Master server, go to the ZooKeeper node: /control-tower/execution/active-instances/:
1. Log in to the Master server with the installer.
2. On the Master server, go to INSTALL_DIR/zookeeper/bin, and execute the ./zkCli.sh script to access the ZooKeeper's interface.
3. In ZooKeeper, execute the following command to view existing ZK nodes:
```
ls /control-tower/execution/active-instances 
```
Wait until all nodes inside the path have status PURGED:
1. In ZooKeeper, execute the get /control-tower/execution/active-instances/NODE_ID command to view the existing ZK nodes status.
2. Wait until all nodes inside the path have the status PURGED.
Recheck the status:
1. If after 30 minutes, you see a node with value PAUSE/PAUSED, copy the UUID (node name) and paste it into CT UI to review the BP it belongs to.
2. Open any BP in Workflow Designer and replace the host UUID with the UUID from the ZK node, containing the status PAUSE/PAUSED.
3. If all nodes have status PURGED, or the BPs that contain only Manual Tasks have the PAUSED status, go to the next step.
4. In ZK path /control-tower/execution/active-instances, delete all nodes by running the command delete /control-tower/execution/activeinstances/NODE_ID for all existing nodes one by one.

Patch Linux servers

Stop components

Run the following commands to stop all Product components on Linux machines:

Quick setup
Standard setup

On the MASTER1 server, run the following commands:

$ cd PACKAGE_DIR
$ export ANSIBLE_PRIVATE_KEY_FILE="</path_to_ssh_key>"   
# Path to SSH key for connection to all Linux servers as WFUSER
$ export ANSIBLE_REMOTE_USER="WFUSER" 
# WFUSER provided in config.yml
$ export ANSIBLE_VAULT_PASS="<ansible_vault_password>"   
# Password for decryption of config.yml. If you don't set this variable, you'll be prompted for password during setup.

On all Master servers, run the following commands:

$ wfmanager stop marathon-apps
$ cd /opt/workfusion/wf_installer 
$ ./install.sh stop_services full

On the MASTER1 server, run the command:
```
```bash
wfmanager stop marathon-apps
```
```
On each Linux server, starting with the MASTER ones, run the command:
```
wfmanager stop all
```

Back up workers

Remove workers from the file system in the shared directory /vds-data. For that, on the MASTER1 server, run the commands:
```
$ cd /opt/workfusion/vds-data/workers/
$ find . -name *.jar ! -iname fre* | xargs rm
```
caution
Do not remove JARs from /vds-data via this command: $ find . -name '*.jar' | xargs rm It removes FRE11 and FRE12 JAR files too, and OCR is unable to get license details. To exclude FRE libs, use the following command:
$ find . -name *.jar ! -iname fre* | xargs rm
If you previously run $ find . -name '*.jar' | xargs rm, perform the following steps to fix the issue:
1. Copy FRE JARs to /vds-data manually:
cp /opt/workfusion/wf_installer/sources/frengine-linux-11*.jar /opt/workfusion/vds-data/workers/app/com.workfusion.ocr/ocr-worker/OCR_WORKER_VERSION/lib11/ cp /opt/workfusion/wf_installer/sources/frengine-linux-12*.jar /opt/workfusion/vds-data/workers/app/com.workfusion.ocr/ocr-worker/OCR_WORKER_VERSION/lib12/
1. Kill OCR workers in Marathon UI.

Back up all worker configuration files. For that, on the MASTER1 server, run the commands:

$ cd /opt/workfusion/vds-data/workers

# This command finds all worker YML configs to view them:
$ find . -iname "worker*.yml"

# This command finds and makes backups of all worker YML config. For example, it will copy worker.yml to worker.yml_BAK in the same directory)
$ find . -name 'worker*.yml' -type f | while read NAME ; do /bin/cp -f "${NAME}" "${NAME}_BAK" ; done

Apply patch

To apply the patch:

On each Linux server, download and extract the logpresso-log4j2-scan utility:

$ cd /opt/workfusion/

$ curl -o 'logpresso-log4j2-scan-3.0.1-linux.tar.gz' 'https://workfusion-installer.s3.amazonaws.com/blobs/binaries/log4j-fix-tool/logpresso-log4j2-scan-3.0.1-linux.tar.gz?AWSAccessKeyId=AKIA47BRAXGS4S3AB3U4&Expires=1755445194&Signature=%2FNFufbzxcpQGP0E1uHQ52Jv05G0%3D'

$ tar xzf logpresso-log4j2-scan-3.0.1-linux.tar.gz

On each Linux server, execute the following command to apply the patch.
note
- If you have previously extracted the Product installer into the installation directory (for example, /opt/workfusion/wf_installer), you may exclude the directory with the installer from the scan path by adding the respective --exclude argument to the below command (for example, --exclude /opt/workfusion/wf_installer).
- If you’re patching the HA environment, add the following arguments when running the patch on MASTER2 and MASTER3 servers: --exclude /opt/workfusion/shared --exclude /opt/workfusion/vds-data.
- Regardless of whether it is HA mode or non-HA, add the following arguments when running the command on the AGENT servers: --exclude /opt/workfusion/vds-data.
See the example sequence of commands on a typical HA environment:
# On MASTER1: ./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/wf_installer /opt/workfusion # On MASTER2 and MASTER3: ./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/shared --exclude /opt/workfusion/vds-data --exclude /opt/workfusion/wf_installer /opt/workfusion # On AGENTs: ./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/vds-data --exclude /opt/workfusion/wf_installer /opt/workfusion
```
./log4j2-scan --debug --fix /opt/workfusion/ --report-csv --report-path ./log4j2-scan-report.csv
```
Start all Product components by running the following commands in the stated order:
1. On each Linux server, starting with the MASTER ones, run the command:
```
wfmanager start all
```
2. On the MASTER1 server, run the command:
```
wfmanager start marathon-apps
```

In Nexus repositories, refresh checksums for modified artifacts. For that, on the MASTER1 server, execute the commands. Ensure to replace NEXUS_LB_HOSTNAME and NEXUS_ADMIN_PASS variables with actual values for your environment.

note

The following requests do not remove any data from Nexus. Thus the “DELETE” HTTP method should not be of concern. That's how Nexus API expects the request to rebuild its metadata.

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/wf-archetypes-releases/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/wf-archetypes-releases/content/wf-archetypes-releases/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/wf-archetypes-releases/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/wf-dependencies/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/wf-dependencies/content/wf-dependencies/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/wf-dependencies/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/workers/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/workers/content/workers/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/workers/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/ml-sdk/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/ml-sdk/content/ml-sdk/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/ml-sdk/content

Patch Windows servers

To apply the patch on the Windows servers, perform the following actions:

Ensure that you are logged into the RPA and Analytics (BI) machines via RDP as a member of the Administrators group.
On the RPA server, stop the WFSvc service.
For Work.AI v10.2.8, on the Analytics (BI) server, stop the Tableau services via the service.msc Windows Services Manager.
On the RPA and BI servers, download and extract the logpresso-log4j2-scan utility.
Open CMD as Administrator and run the logpresso utility against the RPA installation directory and BI installation directory, respectively:
```
log4j2-scan.exe --debug --fix --report-csv --report-path c:\log4j2-scan-report.csv c:\workfusion\
```
You may see in your report that a few files are not mitigated entirely in your first run on the BI server. To fix the files, in your CMD opened under Administrator, run the below command with different file report output against RPA and BI installation directories, respectively.
```
log4j2-scan.exe --debug --fix --report-csv --report-path c:\log4j2-scan-report1.csv c:\workfusion\
```

Restore worker configs

If you had customized worker configuration files on your environment, run your BPs to ensure that new workers are deployed with default configs. Then, restore the backed-up worker configs.

It is OK if some business processes fail before restoring worker configs. If needed, on the MASTER1 server, run the following commands.

$ cd /opt/workfusion/vds-data/workers

# the command restores worker configs from backups
find . -name 'worker*.yml_BAK' -type f | while read NAME ; do /bin/cp -f "${NAME}" "${NAME%_BAK}" ; done

Apply patch​

Prepare for patching​

Stop Business Processes​

Pause Business Processes​

Patch Linux servers​

Stop components​

Back up workers​

Apply patch​

Patch Windows servers​

Restore worker configs​

Apply patch

Prepare for patching

Stop Business Processes

Pause Business Processes

Patch Linux servers

Stop components

Back up workers

Apply patch

Patch Windows servers

Restore worker configs