Edit

Generate features for Data Cleansing Tool

The Data Cleansing Tool is used for detecting documents with noise (poorly tagged, with wrong labels, and so on) in a training set.

The tool needs features generated by a model in a unified format.

Since version 10.2.0.3, all out-of-the-box models have the Data Cleansing Mode for training. In this mode, the model is not trained. Instead, features.zip is generated with unified features in the output for the Data Cleansing Tool.

To enable this mode for model training, add the DATA_CLEANSING_MODE parameter with the true value. See the example below:

  Map<String, Object> params = new HashMap<>();
  params.put(ConfigurationConstants.DATA_CLEANSING_MODE, true);

  LocalTrainingConfiguration configuration = LocalTrainingConfiguration.builder()
    ...
    .parameters(params)
    ...
    .build();