Customize Python code

The customization affects the pipeline.py file located on the following path: src/main/python/package/automl/ml/lib/{libName}/pipeline.py. The file is part of a project template and serves as an entry point for integrating Python models.

Developers are required to implement the _fit method to allow models to learn from input data sets and the _predict method to execute models.

Change _fit method

The method is intended to train models. It gets the input cxt(context) parameter containing path configurations. If you have a custom training logic, you can add it to the method.

If you already have a trained model, it is recommended to implement a fake _fit method. It returns the same result for all documents. This helps you to get a trained model artifact with all essential files and a folder structure for integration without any manual effort.

See the fake training method example below for a Classification model:

def _fit(self, ctx):
    """ Fit pipeline
    Parameters
    ----------
    ctx : ClassificationFitContext
        context of the fit flow
    """
    logger.info(u'Process fitting pipeline')
    # process training set and train model
    # prepare eval metadata and eval results
    # save trained model
    test_set_path = ctx.test_set_path
    for filename in os.listdir(test_set_path):
        logger.info(u'\x1b[34m filename: %s \x1b[0m', filename)
        with open(os.path.join(test_set_path, filename)) as f:
            classification_document = ClassificationDocument(filename, labels=[Label(0, 0, "1", 1)])
            self._eval_metadata.add(classification_document)
            result = EvaluationResult(doc_id=filename,
                                      extracted="1",
                                      extracted_value=1,
                                      gold="1",
                                      gold_value=1, scores={'0': 0, '1': 1},
                                      attrs={})
            self._eval_results.add(result)

Modify _predict method

The _predict method is used for execution. It gets the document and ctx parameters as input, as well as document id, and text.

To implement a custom prediction logic, add it to the _predict(self, document) method. The method returns ClassificationDocument for a Classification model and brings its output to the following format:

return ClassificationDocument(document.id, labels=[Label(0, 0, u'some_label', 1)])
                      #Label(strartPosition, endPosition, extractedField, score)

It is recommended to use pipeline.py as an integration layer and implement all model logic in a separate class as described in the next section.

Create wrapper class

It is not recommended to implement all logic in a pipeline. The best practice is to implement a separate model class to be used within the pipeline. For the class, employ the following methods: __init__(self, ctx), predict(self, document), fit(self).

Change __init__ method in wrapper

In the __init__ method, init all necessary information for model training and execution, for instance, Python models to be executed.

Below, you can see an example of the __init__ method implemented for a class:

def __init__(self, ctx):
    self.ctx = ctx
    self.resources_path = os.path.join(self.ctx.model_path, os.pardir, 'resources', 'models')
 
    ###############################################################################
    # LOAD CLASSIFIERS
    ###############################################################################
 
    self.logreg = load(os.path.join(self.resources_path, 'clf', 'logreg.model'))
    self.rfc = load(os.path.join(self.resources_path, 'clf', 'rfc.model'))
    self.ada = load(os.path.join(self.resources_path, 'clf', 'ada.model'))
    self.svc = load(os.path.join(self.resources_path, 'clf', 'svc.model'))
    self.ensemble = load(os.path.join(self.resources_path, 'clf', 'ensemble.model'))
 
    with open(os.path.join(self.resources_path, 'pred_score_cutoffs.pkl'), 'rb') as f:
        self.cutoffs = pickle.load(f)

Change _predict and _fit methods in wrapper

To change _predict and _fit methods in the wrapper, follow the steps below:

1. Move the logic for the _predict and _fit methods from the pipeline to the wrapper methods.

2. Move the init commands to the __init__ method as described above.

Init wrapper and invoke its methods

To init a wrapper and invoke its methods, follow the steps below:

1. Init the model in the _predict and _fit methods inside the pipeline. Initialize the model as a class attribute.

2. Invoke the methods within the wrapper.

Below, you can see an example of model initialization in the methods:

def _predict(self, document, ctx):
    if self.model is None:
        self.model = Model(ctx)
 
    result = self.model.predict(document)
 
    return ClassificationDocument(document.id, labels=[Label(0, len(document.text), result, 1)])

tip

Be careful not to break the class structure while changing the pipeline. The __init__, _fit, _predict, _get_eval_metadata, _get_eval_results methods must be in the same class.