Prepare environment

Legend

TENANT_ID: Tenant name; must contain only alphanumeric symbols, for example, WiseRealm
TENANT_ID_LC: Tenant name in lowercase
TENANT_ID_UC: Tenant name in uppercase
INSTALL_DIR: directory where all components are installed, for example, /opt/workfusion
IMPORT_DIR: directory scanned for realm files, for example, INSTALL_DIR/keycloak/realm
CONNECTOR_DIR: home directory of the Input REST Connector application on the Agent server, for example, INSTALL_DIR/input-connector-rest
MARATHON_DIR: home directory of the Marathon application on the Master server, for example, INSTALL_DIR/marathon
NGINX_DIR: home directory of the Nginx application on the Agent server, for example, INSTALL_DIR/nginx
HAPROXY_DIR: home directory of the HAProxy application on the Master server, for example, INSTALL_DIR/haproxy
CONNECTOR_PORT: port used by Input REST Connector to retrieve requests
CT_LB_HOST: lb hostname of Control Tower or Input REST Connector, for example, multi-realm-dw-rest-api-wfaw-10081-workfusion-lb1.wflab.io
AGENT_HOST: hostname of the Agent server, for example, multi-realm-dw-rest-api-wfaw-10081-agent1.wflab.io

Perform manual steps

To prepare your environment for the multi-realm usage, perform the following setup.

Prepare Keycloak

To set up Keycloak, do as follows:

Configure a client secret for the Control Tower application:
- keycloak.realm.TENANT_ID.ct.sso.oidc.client.secret, for example, keycloak.realm.WiseRealm.ct.sso.oidc.client.secret
- keycloak.realm.TENANT_ID.ws.sso.oidc.client.secret
  
  note
  
  For WorkSpace, use the loader.sh script to upload the property for the WFAGENT user, as the property name is not in the whitelist for WorkSpace.
Prepare a realm:
1. Replace ID and name in the realm JSON file to TENANT_NAME.
2. Replace all WorkfusionRealm UUIDs with new ones. Mind that Keycloak doesn't perform auto-replacement, and you should do it manually.
3. Replace client secrets of WorkSpace and Control Tower clients to new ones. For example, for a Control Tower client, replace ${vault.ct.sso.oidc.client.secret} to ${vault.keycloak.realm.TENANT_ID.ct.sso.oidc.client.secret}.
4. Put the realm into IMPORT_DIR.
Prepare the supervisor startup script to support import from the realm:
1. In /opt/workfusion/supervisord/apps/keycloak.ini, add the following lines for the standalone.sh start parameters:
```
-Dkeycloak.migration.action=import \
-Dkeycloak.migration.provider=dir \
-Dkeycloak.migration.dir=IMPORT_DIR \
-Dkeycloak.migration.strategy=IGNORE_EXISTING
```
1. To update, run the following command:
```
wfmanager update
```
1. Restart Keycloak:
```
wfmanager restart keycloak
```
Add the realm to the keycloak.realms.additional secret property, for example, keycloak.realms.additional=IMPORTED_EARLIER_FOR_EXAMPLE,TENANT_ID.

Prepare Marathon

To configure the Marathon application, do as follows:

Make a copy of run-input-connector-rest.sh on each Agent server to a new file, for example, CONNECTOR_DIR/run-input-connector-rest-TENANT_ID.sh.

In the script, specify the realm and server port and customize output file locations for logs and other data, for example:

#!/bin/bash

REALM_NAME=TENANT_ID
APP_PORT=CONNECTOR_PORT
set -e
echo -e "\n\n$(date) input-connector-rest STARTUP INITIATED" >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log

source /opt/workfusion/environment.sh

check-vault.sh    >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
check-keycloak.sh >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log

echo "$(date) Trying to read variables from Vault..." >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
echo "$(date) Trying to get VAULT_TOKEN..."           >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
export VAULT_TOKEN=$(get-vault-token.sh)

echo "$(date) Trying to read rabbitmq_bep_pass from Vault..." >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
export SPRING_RABBITMQ_PASSWORD=$(get-vault-secret.sh workfusion_ansible_secrets/rabbitmq_bep_pass)

unset VAULT_TOKEN

echo "$(date) Dependencies check passed. Starting..."      >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
echo "$(date) cd /opt/workfusion/input-connector-rest"      >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
echo "$(date) export SPRING_RABBITMQ_PASSWORD=..."         >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
echo "$(date) exec java -jar input-connector-rest.jar" >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log

cd /opt/workfusion/input-connector-rest
exec java -jar input-connector-rest.jar --server.port=$APP_PORT --icr.sso.oidc.realm-name=$REALM_NAME >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log

Add the Marathon application on the Master server.

Copy the default input-connector-rest.json file to input-connector-rest-TENANT_ID_LC.json, for example, MARATHON_DIR/apps/input-connector-rest-TENANT_ID_LC.json.
Fix the ID with the realm prefix (should be in lowercase), for example, /ia-cloud/input-connector-rest-TENANT_ID_LC.

Fix cmd to a customized script, for example:

exec CONNECTOR_DIR/run-input-connector-rest-TENANT_ID.sh

Specify a health check endpoint, for example:

curl -s -w '%{http_code}' -o /dev/null http://127.0.0.1:CONNECTOR_PORT/actuator/health | grep 200

Deploy to Marathon. In the manual mode, copy the JSON file and put it into the Marathon UI > Create application. For example, a JSON file for the UI can look as follows:

{
"id": "/ia-cloud/input-connector-rest-wiserealm",
"cmd": "exec /opt/workfusion/input-connector-rest/run-input-connector-rest-wiserealm.sh",
"cpus": 0,
"mem": 512,
"disk": 0,
"instances": 1,
"constraints": [
    [
    "bep-worker",
    "CLUSTER",
    "true"
    ],
    [
    "hostname",
    "UNIQUE"
    ]
],
"acceptedResourceRoles": [
    "*"
],
"healthChecks": [
    {
    "command": {
        "value": "curl -s -w '%{http_code}' -o /dev/null http://127.0.0.1:8082/actuator/health | grep 200"
    },
    "gracePeriodSeconds": 300,
    "intervalSeconds": 60,
    "maxConsecutiveFailures": 0,
    "protocol": "COMMAND",
    "ipProtocol": "IPv4",
    "timeoutSeconds": 40
    }
],
"portDefinitions": [],
"taskKillGracePeriodSeconds": 20,
"killSelection": "YOUNGEST_FIRST",
"unreachableStrategy": {
    "inactiveAfterSeconds": 0,
    "expunceAfterSeconds": 0
},
"upgradeStrategy": {
    "maximumOverCapacity": 0,
    "minimumHealthCapacity": 0
}
}

Prepare Nginx

To prepare Nginx, perform the following steps:

For each Agent server, add the following locations to NGINX_DIR/sites/input-connector-rest.conf:

  location = /TENANT_ID/input-connector-rest {
    rewrite ^ /TENANT_ID/input-connector-rest/;
  }

  location /TENANT_ID/input-connector-rest/ {
    proxy_pass            http://127.0.0.1:<connector_port>/;
    proxy_redirect        off;
    proxy_send_timeout    600;
    proxy_read_timeout    600;
    proxy_connect_timeout 600;
    send_timeout          600;
    # Increase client_max_body_size for large reguests to automl-model-management-service
    client_max_body_size 1024m;
   }

Restart Nginx:
```
wfmanager restart nginx
```

Prepare HAProxy

To set up HAProxy, add the following sections to HAPROXY_DIR/conf/haproxy.conf on the Master server:

Add ACL:

acl to_input_connector_rest_TENANT_ID_LC         path_beg -i /TENANT_ID/input-connector-rest

Exclude the endpoint from the Control Tower backend to ACL mapping by ! to_input_connector_rest_TENANT_ID_LC.

For example:

if to_ct ! to_ags ! to_ams ! to_abs ! to_amms ! to_wms ! to_tds ! to_sqc ! to_dataset_management ! to_dataset_management_internal ! to_labeling_service ! to_zo
onavigator ! to_manual_task_renderer ! to_task_designer ! to_input_connector_rest ! to_input_connector_rest_TENANT_ID_LC

Add a separate backend to ACL mapping:

use_backend INPUT_CONNECTOR_REST_TENANT_ID_UC         if to_input_connector_rest_TENANT_ID_LC to_ct

Add the backend:

# INPUT CONNECTOR REST TENANT_ID_UC
backend INPUT_CONNECTOR_REST_TENANT_ID_UC
    mode http
    option httpchk GET /TENANT_ID/input-connector-rest/actuator/health HTTP/1.1\r\nHost:\ CT_LB_HOST
    http-check expect status 200
        # describe this mapping for each Agent server
        server AGENT_HOST AGENT_HOST:8443 ssl verify none on-marked-down shutdown-sessions check check-ssl port 8443 inter 20000 rise 1 fall 1

Restart HAProxy:
```
wfmanager restart haproxy
```

Modify Input Connector configuration

In the manual steps above, for Input REST Connector, you create separate scripts and pass parameters in the command line of Java startup.

You can also use spring profiles to determine parameters for each application, for example:

Create the base application.yml file:

spring:
  codec:
    max-in-memory-size: 2MB
  rabbitmq:
    addresses: 'multi-realm-dw-rest-api-wfaw-10081-master1.wflab.io:5672'
    virtual-host: bep
    username: bep

server:
  address: 127.0.0.1
# port should be specified by profile properties file  
#  port: 8081

connector:
  response-timeout: 60m
  record-queue: ct-record-queue

icr:
  sso:
    oidc:
      enabled: true
      auth-url: 'https://multi-realm-dw-rest-api-wfaw-10081-auth-lb1.wflab.io/auth'
#      realm should be specified by profile properties file
#      realm-name: WorkfusionRealm
      client-id: wf-control-tower
      online-cache-size: 1000
      online-cache-ttl-minutes: 3
      public-key-update-millis: 120000
      update-not-before-millis: 120000

For each new Tenant, create a separate application-TENANT_ID.yml file:

server:
  port: CONNECTOR_PORT
icr:
  sso:
    oidc:
      realm-name: TENANT_ID

Run run-input-connector-rest.sh. Mind that only one script is used for all Connector instances.

#!/bin/bash

TENANT_PROFILE=$1
set -e
echo -e "\n\n$(date) input-connector-rest STARTUP INITIATED" >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log

source /opt/workfusion/environment.sh

check-vault.sh    >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
check-keycloak.sh >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log

echo "$(date) Trying to read variables from Vault..." >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
echo "$(date) Trying to get VAULT_TOKEN..."           >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
export VAULT_TOKEN=$(get-vault-token.sh)

echo "$(date) Trying to read rabbitmq_bep_pass from Vault..." >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
export SPRING_RABBITMQ_PASSWORD=$(get-vault-secret.sh workfusion_ansible_secrets/rabbitmq_bep_pass)

unset VAULT_TOKEN

echo "$(date) Dependencies check passed. Starting..."      >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
echo "$(date) cd /opt/workfusion/input-connector-rest"      >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
echo "$(date) export SPRING_RABBITMQ_PASSWORD=..."         >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
echo "$(date) exec java -jar input-connector-rest.jar" >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log

cd /opt/workfusion/input-connector-rest
exec java -jar input-connector-rest.jar --spring.profiles.active=$TENANT_PROFILE >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log

Run the following command for the Marathon application:

exec /opt/workfusion/input-connector-rest/run-input-connector-rest.sh TENANT_ID

With such modifications, perform the following steps for each new Tenant:

Create a new application-TENANT_ID.yml file.
Pass TENANT_ID in cmd of the Marathon application.

tip

For Tenant configuration, see Add Tenant.

WorkFusion Documentation

10.2.7

Hardware

Install Linux components

Keycloak administrator guide

Legend

Perform manual steps

Prepare Keycloak

note

Prepare Marathon

Prepare Nginx

Prepare HAProxy

Modify Input Connector configuration

tip