Prepare environment

Legend

• TENANT_ID: Tenant name; must contain only alphanumeric symbols, for example, WiseRealm
• TENANT_ID_LC: Tenant name in lowercase
• TENANT_ID_UC: Tenant name in uppercase
• INSTALL_DIR: directory where all components are installed, for example, /opt/workfusion
• IMPORT_DIR: directory scanned for realm files, for example, INSTALL_DIR/keycloak/realm
• CONNECTOR_DIR: home directory of the Input REST Connector application on the Agent server, for example, INSTALL_DIR/input-connector-rest
• MARATHON_DIR: home directory of the Marathon application on the Master server, for example, INSTALL_DIR/marathon
• NGINX_DIR: home directory of the Nginx application on the Agent server, for example, INSTALL_DIR/nginx
• HAPROXY_DIR: home directory of the HAProxy application on the Master server, for example, INSTALL_DIR/haproxy
• CONNECTOR_PORT: port used by Input REST Connector to retrieve requests
• CT_LB_HOST: lb hostname of Control Tower or Input REST Connector, for example, multi-realm-dw-rest-api-wfaw-10081-workfusion-lb1.wflab.io
• AGENT_HOST: hostname of the Agent server, for example, multi-realm-dw-rest-api-wfaw-10081-agent1.wflab.io

Perform manual steps

To prepare your environment for the multi-realm usage, perform the following setup.

Prepare Keycloak

To set up Keycloak, do as follows:

1. Configure a client secret for the Control Tower application:

   • keycloak.realm.TENANT_ID.ct.sso.oidc.client.secret, for example, keycloak.realm.WiseRealm.ct.sso.oidc.client.secret

   • keycloak.realm.TENANT_ID.ws.sso.oidc.client.secret

     note

     For Workspace, use the loader.sh script to upload the property for the WFAGENT user, as the property name is not in the whitelist for Workspace.

2. Prepare a realm:

   1. Replace ID and name in the realm JSON file to TENANT_NAME.
   2. Replace all WorkfusionRealm UUIDs with new ones. Mind that Keycloak doesn't perform auto-replacement, and you should do it manually.
   3. Replace client secrets of Workspace and Control Tower clients to new ones. For example, for a Control Tower client, replace ${vault.ct.sso.oidc.client.secret} to ${vault.keycloak.realm.TENANT_ID.ct.sso.oidc.client.secret}.
   4. Put the realm into IMPORT_DIR.

3. Prepare the supervisor startup script to support import from the realm:

   1. In /opt/workfusion/supervisord/apps/keycloak.ini, add the following lines for the standalone.sh start parameters:

      -Dkeycloak.migration.action=import \
      -Dkeycloak.migration.provider=dir \
      -Dkeycloak.migration.dir=IMPORT_DIR \
      -Dkeycloak.migration.strategy=IGNORE_EXISTING

   2. To update, run the following command:

      wfmanager update

   3. Restart Keycloak:

      wfmanager restart keycloak

4. Add the realm to the keycloak.realms.additional secret property, for example, keycloak.realms.additional=IMPORTED_EARLIER_FOR_EXAMPLE,TENANT_ID.

Prepare Marathon

To configure the Marathon application, do as follows:

1. Make a copy of run-input-connector-rest.sh on each Agent server to a new file, for example, CONNECTOR_DIR/run-input-connector-rest-TENANT_ID.sh.

   In the script, specify the realm and server port and customize output file locations for logs and other data, for example:

   #!/bin/bash
   
   REALM_NAME=TENANT_ID
   APP_PORT=CONNECTOR_PORT
   set -e
   echo -e "\n\n$(date) input-connector-rest STARTUP INITIATED" >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   
   source /opt/workfusion/environment.sh
   
   check-vault.sh    >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   check-keycloak.sh >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   
   echo "$(date) Trying to read variables from Vault..." >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   echo "$(date) Trying to get VAULT_TOKEN..."           >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   export VAULT_TOKEN=$(get-vault-token.sh)
   
   echo "$(date) Trying to read rabbitmq_bep_pass from Vault..." >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   export SPRING_RABBITMQ_PASSWORD=$(get-vault-secret.sh workfusion_ansible_secrets/rabbitmq_bep_pass)
   
   unset VAULT_TOKEN
   
   echo "$(date) Dependencies check passed. Starting..."      >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   echo "$(date) cd /opt/workfusion/input-connector-rest"      >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   echo "$(date) export SPRING_RABBITMQ_PASSWORD=..."         >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   echo "$(date) exec java -jar input-connector-rest.jar" >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log
   
   cd /opt/workfusion/input-connector-rest
   exec java -jar input-connector-rest.jar --server.port=$APP_PORT --icr.sso.oidc.realm-name=$REALM_NAME >> /opt/workfusion/logs/input-connector-rest-$REALM_NAME.log

2. Add the Marathon application on the Master server.

   1. Copy the default input-connector-rest.json file to input-connector-rest-TENANT_ID_LC.json, for example, MARATHON_DIR/apps/input-connector-rest-TENANT_ID_LC.json.

   2. Fix the ID with the realm prefix (should be in lowercase), for example, /ia-cloud/input-connector-rest-TENANT_ID_LC.

   3. Fix cmd to a customized script, for example:

      exec CONNECTOR_DIR/run-input-connector-rest-TENANT_ID.sh

   4. Specify a health check endpoint, for example:

      curl -s -w '%{http_code}' -o /dev/null http://127.0.0.1:CONNECTOR_PORT/actuator/health | grep 200

   5. Deploy to Marathon. In the manual mode, copy the JSON file and put it into the Marathon UI > Create application. For example, a JSON file for the UI can look as follows:

      {
      "id": "/ia-cloud/input-connector-rest-wiserealm",
      "cmd": "exec /opt/workfusion/input-connector-rest/run-input-connector-rest-wiserealm.sh",
      "cpus": 0,
      "mem": 512,
      "disk": 0,
      "instances": 1,
      "constraints": [
          [
          "bep-worker",
          "CLUSTER",
          "true"
          ],
          [
          "hostname",
          "UNIQUE"
          ]
      ],
      "acceptedResourceRoles": [
          "*"
      ],
      "healthChecks": [
          {
          "command": {
              "value": "curl -s -w '%{http_code}' -o /dev/null http://127.0.0.1:8082/actuator/health | grep 200"
          },
          "gracePeriodSeconds": 300,
          "intervalSeconds": 60,
          "maxConsecutiveFailures": 0,
          "protocol": "COMMAND",
          "ipProtocol": "IPv4",
          "timeoutSeconds": 40
          }
      ],
      "portDefinitions": [],
      "taskKillGracePeriodSeconds": 20,
      "killSelection": "YOUNGEST_FIRST",
      "unreachableStrategy": {
          "inactiveAfterSeconds": 0,
          "expunceAfterSeconds": 0
      },
      "upgradeStrategy": {
          "maximumOverCapacity": 0,
          "minimumHealthCapacity": 0
      }
      }

Prepare Nginx

To prepare Nginx, perform the following steps:

1. For each Agent server, add the following locations to NGINX_DIR/sites/input-connector-rest.conf:

     location = /TENANT_ID/input-connector-rest {
       rewrite ^ /TENANT_ID/input-connector-rest/;
     }
   
     location /TENANT_ID/input-connector-rest/ {
       proxy_pass            http://127.0.0.1:<connector_port>/;
       proxy_redirect        off;
       proxy_send_timeout    600;
       proxy_read_timeout    600;
       proxy_connect_timeout 600;
       send_timeout          600;
       # Increase client_max_body_size for large reguests to automl-model-management-service
       client_max_body_size 1024m;
      }

2. Restart Nginx:

   wfmanager restart nginx

Prepare HAProxy

To set up HAProxy, add the following sections to HAPROXY_DIR/conf/haproxy.conf on the Master server:

1. Add ACL:

   acl to_input_connector_rest_TENANT_ID_LC         path_beg -i /TENANT_ID/input-connector-rest

2. Exclude the endpoint from the Control Tower backend to ACL mapping by ! to_input_connector_rest_TENANT_ID_LC.

   For example:

   if to_ct ! to_ags ! to_ams ! to_abs ! to_amms ! to_wms ! to_tds ! to_sqc ! to_dataset_management ! to_dataset_management_internal ! to_labeling_service ! to_zo
   onavigator ! to_manual_task_renderer ! to_task_designer ! to_input_connector_rest ! to_input_connector_rest_TENANT_ID_LC

3. Add a separate backend to ACL mapping:

   use_backend INPUT_CONNECTOR_REST_TENANT_ID_UC         if to_input_connector_rest_TENANT_ID_LC to_ct

4. Add the backend:

   # INPUT CONNECTOR REST TENANT_ID_UC
   backend INPUT_CONNECTOR_REST_TENANT_ID_UC
       mode http
       option httpchk GET /TENANT_ID/input-connector-rest/actuator/health HTTP/1.1\r\nHost:\ CT_LB_HOST
       http-check expect status 200
           # describe this mapping for each Agent server
           server AGENT_HOST AGENT_HOST:8443 ssl verify none on-marked-down shutdown-sessions check check-ssl port 8443 inter 20000 rise 1 fall 1

5. Restart HAProxy:

   wfmanager restart haproxy

Modify Input Connector configuration

In the manual steps above, for Input REST Connector, you create separate scripts and pass parameters in the command line of Java startup.

You can also use spring profiles to determine parameters for each application, for example:

1. Create the base application.yml file:

   spring:
     codec:
       max-in-memory-size: 2MB
     rabbitmq:
       addresses: 'multi-realm-dw-rest-api-wfaw-10081-master1.wflab.io:5672'
       virtual-host: bep
       username: bep
   
   server:
     address: 127.0.0.1
   # port should be specified by profile properties file  
   #  port: 8081
   
   connector:
     response-timeout: 60m
     record-queue: ct-record-queue
   
   icr:
     sso:
       oidc:
         enabled: true
         auth-url: 'https://multi-realm-dw-rest-api-wfaw-10081-auth-lb1.wflab.io/auth'
   #      realm should be specified by profile properties file
   #      realm-name: WorkfusionRealm
         client-id: wf-control-tower
         online-cache-size: 1000
         online-cache-ttl-minutes: 3
         public-key-update-millis: 120000
         update-not-before-millis: 120000

2. For each new Tenant, create a separate application-TENANT_ID.yml file:

   server:
     port: CONNECTOR_PORT
   icr:
     sso:
       oidc:
         realm-name: TENANT_ID

3. Run run-input-connector-rest.sh. Mind that only one script is used for all Connector instances.

   #!/bin/bash
   
   TENANT_PROFILE=$1
   set -e
   echo -e "\n\n$(date) input-connector-rest STARTUP INITIATED" >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   
   source /opt/workfusion/environment.sh
   
   check-vault.sh    >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   check-keycloak.sh >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   
   echo "$(date) Trying to read variables from Vault..." >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   echo "$(date) Trying to get VAULT_TOKEN..."           >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   export VAULT_TOKEN=$(get-vault-token.sh)
   
   echo "$(date) Trying to read rabbitmq_bep_pass from Vault..." >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   export SPRING_RABBITMQ_PASSWORD=$(get-vault-secret.sh workfusion_ansible_secrets/rabbitmq_bep_pass)
   
   unset VAULT_TOKEN
   
   echo "$(date) Dependencies check passed. Starting..."      >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   echo "$(date) cd /opt/workfusion/input-connector-rest"      >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   echo "$(date) export SPRING_RABBITMQ_PASSWORD=..."         >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   echo "$(date) exec java -jar input-connector-rest.jar" >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log
   
   cd /opt/workfusion/input-connector-rest
   exec java -jar input-connector-rest.jar --spring.profiles.active=$TENANT_PROFILE >> /opt/workfusion/logs/input-connector-rest-$TENANT_PROFILE.log

4. Run the following command for the Marathon application:

   exec /opt/workfusion/input-connector-rest/run-input-connector-rest.sh TENANT_ID

With such modifications, perform the following steps for each new Tenant:

1. Create a new application-TENANT_ID.yml file.
2. Pass TENANT_ID in cmd of the Marathon application.

tip

For Tenant configuration, see Add Tenant.