10.0

Edit

Security advisories

warning

This document was updated on December 22, 2021.

If you have applied any fixes before that date, study the following guide carefully as it may contain critical information regarding the log4j resolution.

The IA Cloud Enterprise releases include the log4j v2.17.0 update. If you have an immediate concern about this vulnerability, you can use the following remediation in the meantime.

Apply patch

To fix the vulnerability, you must apply the patch to all Linux and Windows servers.

note

The following instruction is a subject for frequent updates. For example, the link to the logpresso-log4j2-scan utility may be updated.

This guide uses /opt/workfusion as the installation directory. Adjust it to your environment in all places.

Prepare for patching

Before applying the patch, ensure that all BPs are stopped in the Control Tower interface.

Stop Business Processes

To stop all Business Processes, on the APP (or INT, depending on the setup) server, go to ZooKeeper node /control-tower/execution/active-instances/:

  1. Log in to the server with the installer.

  2. In the INSTALL_DIR/zookeeper/bin directory, execute the ./zkCli.sh script to access the ZooKeeper's interface.

  3. In ZooKeeper, execute the following command to view existing ZK nodes:

    ls /control-tower/execution/active-instances

    The command shows a similar output:

If you see an active instance in your result above, log in to Control Tower, go to the active Business Process, and clear any search filter.

If you see any Active Running BP, stop the BP before turning off all the components.

note

If you can’t stop the BP due to pending Manual Tasks in your current BP’s, follow the Pause Business Processes instruction.

Pause Business Processes

To pause a Business Process, do the following:

  1. Log in to Control Tower, and pause all Business Processes from there.

  2. On the APP (or INT, depending on the setup), go to the ZooKeeper node: /control-tower/execution/active-instances/:

    1. Log in to the APP (INT) server with the installer.

    2. On the server, go to INSTALL_DIR/zookeeper/bin, and execute the ./zkCli.sh script to access the ZooKeeper's interface.

    3. In ZooKeeper, execute the following command to view existing ZK nodes:

      ls /control-tower/execution/active-instances

      The command shows a similar output:

  3. Wait until all nodes inside the path have status PURGED:

    1. In ZooKeeper, execute the get /control-tower/execution/active-instances/NODE_ID command to view the existing ZK nodes status.

    2. Wait until all nodes inside the path have the status PURGED.

  4. Recheck the status:

    1. If after 30 minutes, you see a node with value PAUSE/PAUSED, copy the UUID (node name) and paste it into CT UI to review the BP it belongs to.

    2. Open any BP in Workflow Designer and replace the host UUID with the UUID from the ZK node, containing the status PAUSE/PAUSED.

    3. If all nodes have status PURGED, or the BPs that contain only Manual Tasks have the PAUSED status, go to the next step.

    4. In ZK path /control-tower/execution/active-instances, delete all nodes by running the command delete /control-tower/execution/activeinstances/NODE_ID for all existing nodes one by one.

Patch Linux servers

Stop components

Run the following commands to stop all Product components on Linux machines:

  • For the single-point setup: on the APP/Integration server, run the command:

    ./install.sh stop_services full

  • For the multi-point setup: on each server at a time in the order OCR, BEP Agents, BEP Master, APP, and INT, run the following command:

    note

    If your setup is configured for the basic topology, run the command only on OCR, BEP Agents, and APP.

    wfmanager stop all

Back up workers

  1. Remove workers from the file system in the shared directory /vds-data. For that, on the APP/INT server, run the commands:

    $ cd /opt/workfusion/vds-data/workers/
$ find . -name *.jar ! -iname fre* | xargs rm
    caution

    Do not remove JARs from /vds-data via this command: $ find . -name '*.jar' | xargs rm It removes FRE11 and FRE12 JAR files too, and OCR is unable to get license details. To exclude FRE libs, use the following command:

    $ find . -name *.jar ! -iname fre* | xargs rm

    If you previously run $ find . -name '*.jar' | xargs rm, perform the following steps to fix the issue:

    1. Copy FRE JARs to /vds-data manually:
    cp /opt/workfusion/wf_installer/sources/frengine-linux-11*.jar /opt/workfusion/vds-data/workers/app/com.workfusion.ocr/ocr-worker/OCR_WORKER_VERSION/lib11/

cp /opt/workfusion/wf_installer/sources/frengine-linux-12*.jar /opt/workfusion/vds-data/workers/app/com.workfusion.ocr/ocr-worker/OCR_WORKER_VERSION/lib12/
    1. Kill OCR workers in Marathon UI.

  2. Back up all worker configuration files. For that, on the BEP Master and BEP Agent servers, run the commands:

    $ cd /opt/workfusion/vds-data/workers

# This command finds all worker YML configs to view them:
$ find . -iname "worker*.yml"

# This command finds and makes backups of all worker YML config. For example, it will copy worker.yml to worker.yml_BAK in the same directory)
$ find . -name 'worker*.yml' -type f | while read NAME ; do /bin/cp -f "${NAME}" "${NAME}_BAK" ; done

Apply patch

To apply the patch:

  1. On each Linux server, download and extract the logpresso-log4j2-scan utility:

    $ cd /opt/workfusion/

$ curl -o 'logpresso-log4j2-scan-1.6.3-linux.tar.gz' 'https://workfusion-installer.s3.amazonaws.com:443/blobs/artifacts/logpresso-log4j2-scan/logpresso-log4j2-scan-1.6.3-linux.tar.gz?AWSAccessKeyId=AKIAIYJMMPKAN2MQSJWA&Expires=1671197086&Signature=uGoBQAWYTAxov2SILYlsyMB4P78%3D'

$ tar xzf logpresso-log4j2-scan-1.6.3-linux.tar.gz

  2. On each Linux server, execute the following command to apply the patch.

    note

    • If you have previously extracted the Product installer into the installation directory (for example, /opt/workfusion/wf_installer), you may exclude the directory with the installer from the scan path by adding the respective --exclude argument to the below command (for example, --exclude /opt/workfusion/wf_installer).

    • If you’re patching the HA environment, add the following arguments when running the patch on the APP/INT servers:--exclude /opt/workfusion/shared --exclude /opt/workfusion/vds-data.

    • If you’re patching the HA environment, add the following arguments when running the patch on BEP Agents and BEP Master servers: --exclude /opt/workfusion/vds-data.

    • Regardless of whether it is HA or non-HA mode, add the following arguments when running the command on the BEP Agent servers: --exclude /opt/workfusion/vds-data.

    See the example sequence of commands on a typical HA environment:

    # On APP, INT, OCR:
./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/shared --exclude /opt/workfusion/vds-data --exclude /opt/workfusion/wf_installer /opt/workfusion

# OCR:
./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/wf_installer /opt/workfusion

# On BEP Master and BEP Agents: 
./log4j2-scan --debug --fix --report-csv --report-path ./log4j2-scan-report.csv --exclude /opt/workfusion/vds-data --exclude /opt/workfusion/wf_installer /opt/workfusion

    ./log4j2-scan --debug --fix /opt/workfusion/ --report-csv --report-path ./log4j2-scan-report.csv

  3. On each server at a time in the order INT, APP, BEP Master, BEP Agent, run the following command:

    wfmanager start all

  4. In Nexus repositories, refresh checksums for modified artifacts. For that, on the INT server, execute the commands. Ensure to replace NEXUS_LB_HOSTNAME and NEXUS_ADMIN_PASS variables with actual values for your environment.

    note

    The following requests do not remove any data from Nexus, the “DELETE” HTTP method should not be of concern. That's how Nexus API expects request to rebuild its metadata.

    $ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/wf-archetypes-releases/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/wf-archetypes-releases/content/wf-archetypes-releases/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/wf-archetypes-releases/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/wf-dependencies/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/wf-dependencies/content/wf-dependencies/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/wf-dependencies/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/workers/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/workers/content/workers/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/workers/content

$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/repositories/ml-sdk/routing
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/metadata/repositories/ml-sdk/content/ml-sdk/
$ curl -kvL -X DELETE -u 'admin:NEXUS_ADMIN_PASS'  https://NEXUS_LB_HOSTNAME/nexus/service/local/data_index/repositories/ml-sdk/content

Patch Windows servers

To apply the patch on the Windows servers, perform the following actions:

  1. Ensure that you are logged into the RPA and Analytics (BI) machines via RDP as a member of the Administrators group.

  2. On the RPA server, stop the WFSvc service.

  3. On the Analytics (BI) server, stop the Tableau services via the service.msc Windows Services Manager.

    note

    When stopping еру Tableau services, ensure to stop Windows Tableau services. For that open the Services window and stop every Tableau-related service.

  4. On the RPA and BI servers, download and extract the logpresso-log4j2-scan utility.

  5. Open CMD as Administrator and run the logpresso utility against the RPA installation directory and BI installation directory, respectively:

    log4j2-scan.exe --debug --fix --report-csv --report-path c:\log4j2-scan-report.csv c:\workfusion\

    You may see in your report that a few files are not mitigated entirely in your first run on the BI server. To fix the files, in your CMD opened under Administrator, run the below command with different file report output against RPA and BI installation directories, respectively.

    log4j2-scan.exe --debug --fix --report-csv --report-path c:\log4j2-scan-report1.csv c:\workfusion\

Restore worker configs

If you had customized worker configuration files on your environment, run your BPs to ensure that new workers are deployed with default configs. Then, restore the backed-up worker configs.

It is OK if some business processes fail before restoring worker configs. If needed, on the BEP Master or BEP Agent server, run the following commands.

$ cd /opt/workfusion/vds-data/workers

# the command restores worker configs from backups
find . -name 'worker*.yml_BAK' -type f | while read NAME ; do /bin/cp -f "${NAME}" "${NAME%_BAK}" ; done
← OverviewVersion 10.2.7.4 →