Skip to main content
Version: 10.2.8

Extend AutoML SDK with Python classifiers

You can extend AutoML SDK with a set of Python classifiers based on Scikit-learn, MXNet, and so on. The extension allows using ML algorithms implemented in Python with AutoML pipelines.

Here's a list of supported Python classifiers:

Set up development environment​

This section contains instructions on setting up a local 3.7.9+ or 3.10.10+ Python development environment for proper building and running a custom model with Python support.

Install system packages​

Choose a proper instruction to install required system packages (Python, libraries, compiler, and so on), depending on your operating system.

On a Windows machine, follow the steps:

  1. Install Visual Studio 2022 Community Edition.

  2. Download and execute one of the following installer packages:

As the final step, add the path to the installed Python directory of the appropriate version to the PATH environment variable (if there is none yet):

  • Python 3.7.9 <python_installed_dir>
  • Python 3.10.10 <python_installed_dir>

Install Python packages​

Before installing Python packages, quit all instances of the terminal and start a new one or launch your IDE with an integrated terminal.

When you use the classifiers during model training, you may get a warning saying that there is no module named cv2 and torch. If you don't want to see the warning, install opencv-python and torch.

python3 -m pip install opencv-python==4.7.0.72
note

If you're planning to use a virtual environment, activate it before implementing Python classifiers.

To install all required Python packages, follow one of the instructions below, depending on your operating system:

To install all required Python packages on Windows, run one of the following commands, depending on the installed Python version:

  • For Python 3.7.9+:

    python -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.21.6 pandas==1.3.5 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.7 scikit-image==0.19.3 scikit-learn==0.23.1 scipy==1.7.3 six==1.16.0 transformers==4.29.2 vowpalwabbit==8.11.0

  • For Python 3.10.10+:

    python -m pip install --user --use-deprecated=legacy-resolver datasets==2.12.0 numpy==1.24.3 pandas==2.0.2 Pillow==9.5.0 pyarrow==12.0.0 pydantic==1.10.8 scikit-image==0.20.0 scikit-learn==1.2.2 scipy==1.10.1 six==1.16.0 torchvision==0.15.2 transformers==4.29.2 vowpalwabbit==9.8.0

You cannot run DP (dynamic programming) algorithms on Windows as they require the mxnet package that has compatibility issues with the numpy package.

Use virtual environment​

If you want to use a virtual environment, set the environment variable:

AUTOML_PYTHON_BIN=<path_to_python_binary_file>

To set the variable, follow the steps below, depending on your operating system:

  • MacOS: ~/.bash_profile → export <VARIABLE>=<VALUE>
  • Linux: ~/.bashrc → export <VARIABLE>=<VALUE>
  • Windows: My Computer > Properties / Environment variables → <VARIABLE>=<VALUE>

After that, quit and restart the terminal or your IDE.

note

If you use Nvidia GPU with configured CUDA, or Intel CPU with the mkl driver, or both, you can install mxnet-cu92, or mxnet-mkl, or mxnet-cu92mkl instead of the mxnet package.

Implement Python Classifiers​

To implement a Python classifier, use the mlConfig Named Entity in your AutoML SDK configuration.

Here's an example of implementing RandomForestClassifier with default parameters:

@Named("mlConfig")
public Classifier mlConfig() {
    return RandomForestClassifier.builder()
            .build();
}

Here's an example of implementing RandomForestClassifier with custom parameters:

@Named("mlConfig")
public Classifier mlConfig() {
    return RandomForestClassifier.builder()
            .maxDepth(10)
            .estimators(100)
            .maxFeatures(MaxFeaturesType.LOG2)
            .build();
}

Here's an example of GenericPythonClassifier—a generic customizable classifier. It can be used to set up any classifier manually by configuring the parameters and Python model name directly.

@Named("mlConfig")
public Classifier mlConfig() {
    return GenericPythonClassifier.builder()
        .modelName("sklearn.RandomForestClassifier")
        .parameter("max_depth", 10)
        .parameter("n_estimators", 100)
        .parameter("random_state", 123)
        .parameter("verbose", true)
        .build()
}