Deploy Python environment

To set up a custom Python environment in a cluster and enable running a model in it, follow the instructions below.

Set up environment

To set up a custom Python environment in a cluster, act as follows:

1. Prepare Python for your cluster OS (Centos OS, by default) with all dependencies (libraries).

   Alternatively, you can pack the libraries as wheels and add them to your Python model. For detailed instructions, refer to Build custom Python model.

2. Install required Python versions.

   When a cluster runs a new model, it can choose any node based on the resouce allocation strategy. Therefore, install Python on each node, except for Master. The recommended installation path is as follows: /opt/workfusion/python/.

3. Add the py{VERSION}.sh file to /opt/workfusion/python/.

   When the Worker Management Service (WMS) starts, it reads the Python version (27, by default) from the respective py{VERSION}.sh file. The variable reads from the WF_PYTHON_VERSION env variable.

4. In the py{VERSION}.sh file, add the following lines:

   # removing old pythons executables from PATH
   pathremove /opt/workfusion/python/ 
   # adding current python executable to PATH
   pathprepend /opt/workfusion/python/python{VERSION}/bin
   # replacing PYTHONPATH
   export PYTHONPATH=/opt/workfusion/python/python{VERSION}/lib:/opt/workfusion/python/python{VERSION}/lib64
   export PYTHONHOME=/opt/workfusion/python/python{VERSION}

note

In the sample above, customize your path to Python: python{VERSION}. For instance, replace python{VERSION} with python3.7.9.

Make sure to copy the same file on each node to opt/workfusion/python/.

Enable model running with custom environment

To enable running a model in the configured custom Python environment, proceed as follows:

1. Modify worker-*-python.yml.

   The worker-{profile}.yml file (for Python, worker-execution-python.yml) configures Worker run parameters. Modifying the file is essential to change the Python version for a Worker (Python 2.7, by default).

   To modify the file, add the following lines in the end of the file:

   env:
   WF_PYTHON_VERSION: py{VERSION}

   where {VERSION} is the version specified in the name of the py{VERSION}.sh file.

2. Deploy the model to your Control Tower (CT) instance. For instructions, refer to Import model customized with AutoML SDK.

Now, you can run the deployed model the same way as any other one on CT.

Change Python version for specific Worker

The step is optional. To modify the Python version for a specific Worker without changing the worker{profile}.yml, override WF_PYTHON_VERSION in ZooKeeper as described below:

1. Add the following node: config/worker-management-service/bep.wms.worker.execute/<group-id>:<artifact-id>:<profile>/env.parameters.

   For instance: config/worker-management-service/bep.wms.worker.execute/com.workfusion.ocr:ocr-worker:execution-v1/env.parameters.

2. Set data for the node as in the example below:

   WF_PYTHON_VERSION: py{VERSION}

   where py{VERSION} is the Python version you are using.