Edit

Customize Python code

The customization affects the pipeline.py file located at src/main/python/package/automl/ml/lib/{libName}/pipeline.py. The file is part of a project template and serves as an entry point for integrating Python models.

Developers are required to implement the fit method to allow models to learn from input data sets and the predict method to execute models.

Change fit method

The fit method is intended to train models. It gets the input cxt(context) parameter containing path configurations. If you have a custom training logic, you can add it to the method.

If you already have a trained model, it is recommended to implement a fake fit method. It returns the same result for all documents. This helps you to get a trained model artifact with all essential files and a folder structure for integration without any manual effort.

See the fake training method example below for a Classification model:

def _fit(self, ctx):
    """ Fit pipeline
    Parameters
    ----------
    ctx : ClassificationFitContext
        context of the fit flow
    """
    logger.info(u'Process fitting pipeline')
    # process training set and train model
    # prepare eval metadata and eval results
    # save trained model
    test_set_path = ctx.test_set_path
    for filename in os.listdir(test_set_path):
        logger.info(u'\x1b[34m filename: %s \x1b[0m', filename)
        with open(os.path.join(test_set_path, filename)) as f:
            classification_document = ClassificationDocument(filename, labels=[Label(0, 0, "1", 1)])
            self._eval_metadata.add(classification_document)
            result = EvaluationResult(doc_id=filename,
                                      extracted="1",
                                      extracted_value=1,
                                      gold="1",
                                      gold_value=1, scores={'0': 0, '1': 1},
                                      attrs={})
            self._eval_results.add(result)

Modify predict method

The predict method is used for execution. It gets the document and ctx parameters as input, as well as document id, and text.

To implement a custom prediction logic, add it to the _predict(self, document) method. The method returns ClassificationDocument for a Classification model and brings its output to the following format:

return ClassificationDocument(document.id, labels=[Label(0, 0, u'some_label', 1)])
                      #Label(strartPosition, endPosition, extractedField, score)

It is recommended to use pipeline.py as an integration layer and implement all model logic in a separate class as described in the next section.

Create wrapper class

It is not recommended to implement all logic in a pipeline. The best practice is to implement a separate model class to be used within the pipeline. For the class, employ the following methods: __init__(self, ctx), predict(self, document), fit(self).

Change init method in wrapper

In the __init__ method, initialize all necessary information for model training and execution, for instance, Python models to be executed.

Below, you can see an example of the __init__ method implemented for a class:

def __init__(self, ctx):
    self.ctx = ctx
    self.resources_path = os.path.join(self.ctx.model_path, os.pardir, 'resources', 'models')
 
    ###############################################################################
    # LOAD CLASSIFIERS
    ###############################################################################
 
    self.logreg = load(os.path.join(self.resources_path, 'clf', 'logreg.model'))
    self.rfc = load(os.path.join(self.resources_path, 'clf', 'rfc.model'))
    self.ada = load(os.path.join(self.resources_path, 'clf', 'ada.model'))
    self.svc = load(os.path.join(self.resources_path, 'clf', 'svc.model'))
    self.ensemble = load(os.path.join(self.resources_path, 'clf', 'ensemble.model'))
 
    with open(os.path.join(self.resources_path, 'pred_score_cutoffs.pkl'), 'rb') as f:
        self.cutoffs = pickle.load(f)

Change predict and fit methods in wrapper

To change the predict and fit methods in the wrapper, do the following:

1. Move the logic for the _predict and _fit methods from the pipeline to the wrapper methods.

2. Move the init commands to the __init__ method as described above.

Init wrapper and invoke its methods

To init a wrapper and invoke its methods, follow the steps below:

1. Init the model in the _predict and _fit methods inside the pipeline. Initialize the model as a class attribute.

2. Invoke the methods within the wrapper.

Below, you can see an example of model initialization in the methods:

def _predict(self, document, ctx):
    if self.model is None:
        self.model = Model(ctx)
 
    result = self.model.predict(document)
 
    return ClassificationDocument(document.id, labels=[Label(0, len(document.text), result, 1)])

tip

Be careful not to break the class structure while changing the pipeline. The __init__,_fit, _predict, _get_eval_metadata, _get_eval_results methods must be in the same class.