ML algorithms

The IE case mathematical description contains the following steps.

Tokenization

The input text is split into minimal units called tokens. Typically, token bounds coincide with word bounds. Each token is a single instance for further classification.

Feature extraction

To train a supervised model, you need a matrix of features (X), where each classification instance is described by a vector of features or independent variables: [x_0, x_1, ..., x_n]. Each instance has a corresponding target label.

The whole space of feature extractors is quite large and includes:

• Features for the token level: word-shape, prefix, suffix, and more.
• Context features: immediate left and right context, cell context, line context, table context, page context, and more.

The feature space size is highly dependent on the size of the documents or dataset. For example, one feature extractor that creates a feature equal to the token's value in lowercase produces N unique features where N is the number of unique words in the training set. On the other hand, the feature extractor for a page number only produces the number of features equal to the maximum amount of pages in a document from the training set.

On average, before applying feature selection or dimensionality reduction techniques, feature space has a size of > 100 K.

AutoML is designed to find optimal representation for each case, for example, a subset of feature extractors that describes the given instances in the best possible way.

Target label representation

As soon as the feature vectors are formed, target variables are specified.

By default, the WorkFusion Information Extraction framework uses the BIOES/BILOU notation for each Named Entity type or field. Instead of classifying each token as belonging to a class, use the following additional tags:

• B-field: the first token of a field consisting of several tokens.
• E-field: the last token of a field consisting of several tokens.
• I-field: a middle token of a field consisting of several tokens (can be several between B and E).
• S-field: a field consisting of a single token.
• O: not a field.

Model training

Multi-class classification is trained to predict each class. By default, it is linear SVM with hinge loss from liblinear implementation, which means the optimization problem looks as follows:

where C > 0 is the regularization parameter. For more details, refer to LIBSVM: A Library for Support Vector Machines | Solving the Quadratic Problems.

note

The selection of loss function and the optimization algorithm is dependent on a particular ML algorithm. For example, if you can use a cross-entropy loss, apply stochastic or mini-batch gradient descent for optimization.

Model prediction

Depending on which ML model is used, you can either get probabilities for each class or some confidence scores, as in the case with SVM, where the confidence score represents the distance to the separating hyperplane. In the latter case, confidence scores are transformed into probabilities using the softmax function over the vector containing distances to separating hyperplane for each class as a result of the one-versus-all approach.

You can also use Platt scaling for obtaining probabilities. For more details, refer to LIBSVM: A Library for Support Vector Machines | Probability Estimates. Mind that this option is disabled as the cross-validation involved in Platt scaling is an expensive operation for large datasets. In addition, probability estimates can be inconsistent with the scores, as the argmax of the scores may not be the argmax of the probabilities.

Chunking

After predicting labels for each token, combine them into chunks. Chunks are meaningful pieces of text consisting of one or several words representing the given Named Entity.

To validate results for multiple token entities, an HMM is used with the class probabilities as emission probabilities and precomputed transaction probabilities showing if a certain BIESO tag will be followed by another BIESO tag. This helps to combine long sequences into chunks, where you can have O-s predicted in the middle of a sequence.

Classification

For classification, the approach is identical except that there is no tokenization and chunking. Also, target labels from the target label representation step are used as provided in the training data.