Extend AutoML SDK with Python classifiers · WorkFusion Documentation

You can extend AutoML SDK with a set of Python classifiers based on Scikit-learn, MXNet, and so on. The extension allows using ML algorithms implemented in Python with AutoML pipelines.

Here's a list of supported Python classifiers:

Set up Python development environment

This section contains instructions on setting up a local Python development environment for proper building and running a custom model with Python support.

Install system packages

Choose a proper instruction to install required system packages (Python, libraries, compiler, and so on), depending on your operating system.

Windows

macOS

Linux

On a Windows machine, follow the steps:

Install Visual Studio 2022 Community Edition.
Download and execute one of the following installer packages:
- Python 3.7.9+
- Python 3.10.10+

On a Mac, follow the steps:

Download and execute one of the following installer packages:
- Python 3.7.9+
- Python 3.10.10+
Install the Compiler (GNU Compiler Collection).
```
brew install gcc
```

On a Linux machine, choose one of the following options, depending on the Linux distribution you use.

Fedora

To install Python 3.7.9+, run the following command:

dnf install gcc python3.7.9 python3.7.9-devel libstdc++

To install Python 3.10.10+, execute the following:

dnf install gcc python3.10.10 python3.10.10-devel libstdc++

Ubuntu

To install Python 3.7.9+, run the following command:
```
apt-get install gcc python3.7.9 python3.7.9-dev
```

To install Python 3.10.10+, execute the following:

apt-get install gcc python3.10.10 python3.10.10-dev

As the final step, add the path to the installed Python directory of the appropriate version to the PATH environment variable (if there is none yet):

Python 3.7.9 <python_installed_dir>
Python 3.10.10 <python_installed_dir>

Install Python packages

Before installing Python packages, quit all instances of the terminal and start a new one or launch your IDE with an integrated terminal.

When you use the classifiers during model training, you may get a warning saying that there is no module named cv2 and torch. To avoid the issue, install opencv-python and torch.

note

If you're planning to use a virtual environment, activate it before implementing Python classifiers.

To install all required Python packages, follow one of the instructions below, depending on your operating system:

Windows

Linux or macOS

To install all required Python packages on Windows, run one of the following commands, depending on the installed Python version:

For Python 3.7.9+:

python -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.21.6 pandas==1.3.5 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.7 scikit-image==0.19.3 scikit-learn==0.23.1 scipy==1.7.3 six==1.16.0 transformers==4.29.2 vowpalwabbit==8.11.0

For Python 3.10.10+:

python -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.24.3 pandas==2.0.2 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.8 scikit-image==0.20.0 scikit-learn==1.2.2 scipy==1.10.1 six==1.16.0 torchvision==0.15.2 transformers==4.29.2 vowpalwabbit==9.8.0

You cannot run DP (dynamic programming) algorithms on Windows as they require the mxnet package that has compatibility issues with the numpy package.

To install all required Python packages on Linux or MacOS, run one of the following commands, depending on the installed Python version:

For Python 3.7.9+:

python3 -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.21.6 pandas==1.3.5 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.7 scikit-image==0.19.3 scikit-learn==0.23.1 scipy==1.7.3 six==1.16.0 transformers==4.29.2 vowpalwabbit==8.11.0 mxnet==1.9.1

For Python 3.10.10+:

python3 -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.24.3 pandas==2.0.2 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.8 scikit-image==0.20.0 scikit-learn==1.2.2 scipy==1.10.1 six==1.16.0 torchvision==0.15.2 transformers==4.29.2 vowpalwabbit==9.8.0 mxnet==1.9.1

If you use a virtual environment, remove the --user flag from the above commands.

Use virtual environment

If you want to use a virtual environment, set the environment variable:

AUTOML_PYTHON_BIN=<path_to_python_binary_file>

To set the variable, follow the steps below, depending on your operating system:

MacOS: ~/.bash_profile → export <VARIABLE>=<VALUE>
Linux: ~/.bashrc → export <VARIABLE>=<VALUE>
Windows: My Computer > Properties / Environment variables → <VARIABLE>=<VALUE>

After that, quit and restart the terminal or your IDE.

note

If you use Nvidia GPU with configured CUDA, or Intel CPU with the mkl driver, or both, you can install mxnet-cu92, or mxnet-mkl, or mxnet-cu92mkl instead of the mxnet package.

Implement Python classifiers

To implement a Python classifier, use the mlConfig Named Entity in your AutoML SDK configuration.

Here's an example of implementing RandomForestClassifier with default parameters:

@Named("mlConfig")
public Classifier mlConfig() {
    return RandomForestClassifier.builder()
            .build();
}

Here's an example of implementing RandomForestClassifier with custom parameters:

@Named("mlConfig")
public Classifier mlConfig() {
    return RandomForestClassifier.builder()
            .maxDepth(10)
            .estimators(100)
            .maxFeatures(MaxFeaturesType.LOG2)
            .build();
}

Here's an example of GenericPythonClassifier—a generic customizable classifier. It can be used to set up any classifier manually by configuring the parameters and Python model name directly.

@Named("mlConfig")
public Classifier mlConfig() {
    return GenericPythonClassifier.builder()
        .modelName("sklearn.RandomForestClassifier")
        .parameter("max_depth", 10)
        .parameter("n_estimators", 100)
        .parameter("random_state", 123)
        .parameter("verbose", true)
        .build()
}