Machine learning algorithms

The mathematical description of the information extraction (IE) and classification cases contains the following steps.

Tokenization

The input text is split into minimal units called tokens. Typically, token bounds coincide with word bounds. Each token is a single instance for further classification.

Feature extraction

To train a supervised model, you need a matrix of features (X), where each classification instance is described by a vector of features or independent variables: [x_0, x_1, ..., x_n]. Each instance has a corresponding target label.

The whole space of feature extractors is quite large and includes the following:

• Token-level features: word-shape, prefix, suffix, and more
• Context features: immediate left and right context, cell context, line context, table context, page context, and more

The feature space size is highly dependent on the size of documents or dataset. For example, one feature extractor that creates a feature equal to the token's value in lowercase produces N unique features, where N is the number of unique words in the training set. On the other hand, the feature extractor for a page number only produces the number of features equal to the maximum number of pages in a document from the training set.

On average, before applying feature selection or dimensionality reduction techniques, the feature space has a size of > 100 K.

AutoML is designed to find optimal representation for each case, for example, a subset of feature extractors that describes the given instances in the best possible way.

Target label representation

As soon as the feature vectors are formed, target variables are specified.

By default, the WorkFusion Information Extraction framework uses the BIOES/BILOU notation for each Named Entity type or field. Instead of classifying each token as belonging to a class, use the following additional tags:

• B-field: the first token of a field consisting of several tokens
• E-field: the last token of a field consisting of several tokens
• I-field: a middle token of a field consisting of several tokens (can be several between B and E)
• S-field: a field consisting of a single token
• O: not a field

Model training

Multi-class classification is trained to predict each class. By default, it is a linear Support Vector Machine (SVM) with a hinge loss from the LIBLINEAR implementation, which means the optimization problem looks as follows:

where C > 0 is the regularization parameter. For more details, refer to LIBSVM: A Library for Support Vector Machines | Solving the Quadratic Problems.

note

The selection of the loss function and the optimization algorithm is dependent on a particular machine learning (ML) algorithm. For example, if you can use a cross-entropy loss, apply stochastic or mini-batch gradient descent for optimization.

Model prediction

Depending on which ML model is used, you can either get probabilities for each class or some confidence scores, as in the case with SVM, where the confidence score represents the distance to the separating hyperplane. In the latter case, confidence scores are transformed into probabilities using the softmax function over the vector containing distances to the separating hyperplane for each class as a result of the one-versus-all approach.

You can also use Platt scaling for obtaining probabilities. For more details, refer to LIBSVM: A Library for Support Vector Machines | Probability Estimates. Mind that this option is disabled as the cross-validation involved in Platt scaling is an expensive operation for large datasets. In addition, probability estimates can be inconsistent with the scores as the argmax of the scores may not be the argmax of the probabilities.

Chunking

After predicting labels for each token, combine them into chunks. Chunks are meaningful pieces of text consisting of one or several words representing a given Named Entity.

To validate results for multiple token entities, an HMM is used with the class probabilities as emission probabilities and precomputed transaction probabilities showing if a certain BIESO tag will be followed by another BIESO tag. This helps to combine long sequences into chunks, where you can have O-s predicted in the middle of a sequence.

Classification

For classification, the approach is identical except that there is no tokenization and chunking. Also, target labels from the target label representation step are used as provided in the training data.