Skip to main content
Version: 10.2.8

Feature engineering

Features are individual measurable properties or characteristics of data being observed. Selecting relevant features and deciding how to encode them for a training can have a great impact on model results. The process of feature development is known as feature engineering or feature generation. Simply put, it is a process of manually constructing new attributes from raw data. It involves intelligent combining or splitting the existing raw attributes into new ones which have a higher predictive power.

A particular target data element can have several features tagging it. It is important to select only the features that are relevant to the problem so that the accuracy of the model improves. It also reduces the complexity of the model as we avoid the least significant or unnecessary features.

Feature engineering steps​

  1. Review the training set and define field patterns.
  2. Create a Feature. 
  3. Run training and evaluate results.

Review training set and define field patterns​

As input data you have list of fields that should be extracted and Training Set with already tagged fields. In the example below you can see tagged Invoice and tagged fields that should be extracted: Client Name, Invoice Date, and group of fields Products with fields Item, Description, Quantity, Discount, Tax Rate and Price.

This step is aimed to find out attributes that better differentiate the patterns for each field. Each field should be carefully analyzed to understand its nature (whether it is time, date, email, address etc.) and possible position and context. Context is basically everything that surrounds the tagged piece of text, its position on the page and in some cases its position relative to other important pieces of information. Context is the most important source of features and is used to determine where a gold value should be searched.

Field analysis usually contains the following steps:

  • Tags distribution (how many objects for each field are in the data set?):

    • Check their consistency (for example, in case of Address field) whether all values look similar.
    • Check whether some values are missed (in case of not required fields). This information is useful in understanding which fields are badly represented. In the example below the training set size is 750 documents, but there are only 30 objects of mot_code field. This field is badly represented.

  • Required vs. not required fields.

  • Group values, for example, like table with list of products with their name, quantity, price.

  • Multiple values for one field in the document.

  • Other examples for analysis are total number of tags, frequency of a given tag, tag with the greatest count, the N most common tags and their frequencies, and so on.

  • Define whether word-shape patterns is applied to the field, and how many occurrences of each pattern is met in the documents:

    • Example 1: for field Invoice Number we should pay attention to shape and numeric.
      • Invoice: xx-xxxx: met in 50 cases.
      • Invoice: xx_xxxx: met in 45 cases.
      • Invoice: xx-Xxxx: met in 5 cases.

You can analyse data using various utilities, for example, Analyse Context:

  • Compare a tagged value and a data value and get the context where they are met.
tagged_valuedata_valueContext
EUREuros or signA sentence where this character is met.
  • Get the first element in the line from the left side:

Feature engineering tips​

  1. Start with a minimum set of features:
    • Include features suggested by domain knowledge.
    • Test these out individually, build from the bottom up.
  2. Keep in mind limits on Feature Engineering. Too much features can lead to overfitting: when model will provide good extraction results and statistics on training set but will have bad results on production (new) set. This can be caused by selecting useless features or dependent and correlated features.

Create features​

Having field patterns and context from the previous step we can proceed with creation of features. Generally all features can be split into several types:

Gold Data Specific Features. This defines the features related to the gold value itself.

Examples:

  • The gold value type (price, percentage, address, year, time, date, phone, email, and so on), in case we do not have specific letter pattern for the words. Examples:
    • Identify if the token contains currency characters.
    • Determine whether the last N chars of the token contain digits.
    • Determine if the covered text contains a number, must contain comma or dot for every 1-3 digits "(\s*[€£₤$]\s*)?[0-9])*(?:\D|$)";
    • Presence of hyphen.
  • define whether token correspond to specific letter pattern (capitalization, numeric, length, special chars in between like hyper, dot, and so on). Word shape features are used to represent such patterns by mapping lower-case letters to ‘x’, upper-case to ‘X’, numbers to ’d’, and punctuation. Thus for example A.B.C would map to A.B.C. and AB/01-03 would map to XX/dd-dd. Examples:
    • Determine if the token is a CUSIP code.
    • Determine whether token wordshape corresponds to specific pattern: Xxx_ddddd, xx/xx-ddd-xx, XXXXXX.
  • N-gram is a sequence of words: a 2-gram (or bigram) is a two-word sequence of words like “please click”, or ”click button”, and a 3-gram (or trigram) is a three-word sequence of words like “please click button”, and so on.

Example 1​

In the example below, in case focus element is a number, then feature will be created with a value equals to 1.

Define if the focus element contains a number
import java.util.Collection;
import java.util.Collections;

import com.workfusion.vds.sdk.api.nlp.fe.Feature;
import com.workfusion.vds.sdk.api.nlp.fe.FeatureExtractor;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.FeatureName;
import com.workfusion.vds.sdk.api.nlp.model.Document;
import com.workfusion.vds.sdk.api.nlp.model.Element;

@FeatureName(IsNumberIncludedFE.FEATURE_NAME)
public class IsNumberIncludedFE<T extends Element> implements FeatureExtractor<T> {

    public static final String FEATURE_NAME = "IsNumberIncluded";

    @Override
    public Collection<Feature> extract(Document document, T element) {
        String text = element.getText();
        if (text.matches(".*\\d.*")) {
            // if text contains a digit, return a feture
            return Collections.singletonList(new Feature(FEATURE_NAME, 1));
        }
        // otherwise return empty list
        return Collections.emptyList();
    }

}

Local Context Features: the information around the gold value. Examples:

  • The N tokens, lines, and cells before, after, above, or below the gold value. Examples:
    • Check whether wordshape of neighboring words corresponds to some specific pattern.
    • Define whether number of tokens covered by cell is equal to some specific value.
    • Define whether token is situated in a specifc line of the focus cell.
    • Check if there are keyword matches in the same line in the non-empty block preceding the block covering the focus line.
    • Determine if the numeric value of the focus cell is equal to the cell above.
    • Determine if number of columns from the current cell to the end of the table is equal to N.
    • Determine whether keywords occur in the same column as the focus cell.

Example 2​

The below example shows, that feature will be created with value equal to 1, in case when token, next to target token, in the same cell contains keyword.

Define whether next token in the same cell contains keyword
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import com.workfusion.vds.sdk.api.nlp.fe.Feature;
import com.workfusion.vds.sdk.api.nlp.fe.FeatureExtractor;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.FeatureName;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.Index;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.IndexType;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.Indexes;
import com.workfusion.vds.sdk.api.nlp.model.Cell;
import com.workfusion.vds.sdk.api.nlp.model.Document;
import com.workfusion.vds.sdk.api.nlp.model.Element;
import com.workfusion.vds.sdk.api.nlp.model.Token;

//feature extractor analyze cell->token covered/covering relations, so index will be added.
//BIDIRECTIONAL index type is required, because feature extractors search cells by token, and tokens by cells.
@Indexes()
@FeatureName(NextCellContainsKeyword.FEATURE_NAME)
public class NextCellContainsKeyword<T extends Element> implements FeatureExtractor<T> {

    public static final String FEATURE_NAME = "NextTokenInCellContainsKeyword";

    private String keyword;

    public NextCellContainsKeyword(String keyword) {
        super();
        // specify keyword
        this.keyword = keyword;
    }

    @Override
    public Collection<Feature> extract(Document document, T element) {
        List<Feature> result = new ArrayList<>();
        // find parent cells on current token
        List<Cell> cells = document.findCovering(Cell.class, element);
        if (!cells.isEmpty()) {
            // first cell will be a target cell
            Cell firstCell = cells.stream().findFirst().get();
            // find all tokens inside cell
            Collection<Token> nextTokens = document.findCovered(Token.class, element.getEnd(), firstCell.getEnd());
            nextTokens.forEach(token -> {
                // if one of the token in cell contains a keyword, add a feature
                if (token.getText().contains(keyword)) {
                    Feature feature = new Feature(FEATURE_NAME, 1.0);
                    result.add(feature);
                }
            });
        }
        return result;
    }

}

Global Context Features: the global information such as the location within specific element (in a line, cell, table, block, or page).

Examples:

  • Check whether a token is on the same page with some keywords or NERs.
  • Determine whether any keyword(s) of the user-provided list are found in the blocks to the left of focus.
  • Check whether number of tokens of needed type in all tables is equal to N.
  • Define if token is in the longest row of all the tables in the document.
  • Check whether number of cells in focus table is equal to N.

Example 3​

The following example defines how many times the text from the focus element is met in the Document. The OnDocumentStart** method is used to prepare data for extraction.

Define how many times the target element is met in document
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.workfusion.vds.sdk.api.nlp.annotation.OnDocumentComplete;
import com.workfusion.vds.sdk.api.nlp.annotation.OnDocumentStart;
import com.workfusion.vds.sdk.api.nlp.fe.Feature;
import com.workfusion.vds.sdk.api.nlp.fe.FeatureExtractor;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.FeatureName;
import com.workfusion.vds.sdk.api.nlp.model.Document;
import com.workfusion.vds.sdk.api.nlp.model.Element;

@FeatureName(TextOccurrenceInDocument.FEATURE_NAME)
public class TextOccurrenceInDocument <T extends Element> implements FeatureExtractor<T> {
    public static final String FEATURE_NAME = "ToknenMetInDocument";

    private Map<String, Integer> textOccurrence = new HashMap<>();
    private int totalFocusElementNumber;

    @OnDocumentStart
    public void onStart(Document document, Class<T> focusClass) {
        // find all focus elements (by default focusElement is Token)
        Collection<T> elements = document.findAll(focusClass);
        // store a total number of elements in document. it will be used for feature normalization
        totalFocusElementNumber = elements.size();
        Map<String, Integer> textOccurrence = new HashMap<>();
        // calculate how many times text present into the document
        for (T element : elements) {
            Integer occurrence = textOccurrence.get(element.getText());
            if (occurrence == null) {
                occurrence = 0;
            } else {
                occurrence++;
            }
            textOccurrence.put(element.getText(), occurrence);
        }
    }

    @Override
    public Collection<Feature> extract(Document document, T element) {
        List<Feature> result = new ArrayList<>();
        // get precalculated number of how many times text of element is met in the document
        Integer number = textOccurrence.get(element.getText());
        // it is strongly recommended to use normalized features (values between 0 and 1), divide to the total number of elements in document to get feature value normalized
        double featureValue = (double)number/totalFocusElementNumber;
        // add feature
        result.add(new Feature(FEATURE_NAME, featureValue));
        return result;
    }

    @OnDocumentComplete
    public void complete() {
        // clear map after usage
        textOccurrence.clear();
    }

}

Named Entity Features: whether a word is a part of any relative Named Entities (NERs), the start, middle, or end of NER, the distance to a specific NER.

  • Check if the text of the focus annotation is fully contained within the text of NERs.
  • Check whether distance between token and specific NER equals to N characters.
  • Determine if focus table contains full address (>=1 each of city, state, and postal code NERs).

Example 4​

In the example below, feature will be created with value equal to 1, in case element is covered  by NER of specific type.

Define if target element covered by NER
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

import com.workfusion.vds.sdk.api.nlp.fe.Feature;
import com.workfusion.vds.sdk.api.nlp.fe.FeatureExtractor;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.FeatureName;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.Index;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.IndexType;
import com.workfusion.vds.sdk.api.nlp.fe.annotation.Indexes;
import com.workfusion.vds.sdk.api.nlp.model.Document;
import com.workfusion.vds.sdk.api.nlp.model.Element;
import com.workfusion.vds.sdk.api.nlp.model.NamedEntity;

//feature extractor logic analyze Token->Named Entity relations, to increase performance index is recommended
@Indexes()
@FeatureName(IsNEPresent.FEATURE_NAME)
public class IsNEPresent<T extends Element> implements FeatureExtractor<T> {

    public static final String FEATURE_NAME = "is_named_entity";
    private final String mentionType;

    public IsNEPresent(String mentionType) {
        // specify required named entity type
        this.mentionType = mentionType;
    }

    @Override
    public Collection<Feature> extract(Document document, T element) {
        List<Feature> result = new ArrayList<>();

        // find all named entities inside token
        List<NamedEntity> namedEntity = document.findCovering(NamedEntity.class, element);
        if (namedEntity.stream()
                .filter(n -> mentionType.equalsIgnoreCase(n.getType()))
                .findAny()
                .isPresent()) {
            // if named entity with required type exist, then return a feature
            return Collections.singletonList(new Feature(FEATURE_NAME, 1));
        }
        return Collections.emptyList();
    }
}

All the Features listed above can be used in any combinations.

Run training and evaluate results​

When you added necessary features, then you can start model training for the field. Once training is finished you should evaluate it.

note

Model evaluation should be implemented on unseen data set. Otherwise model evaluation results will be overstated.