Skip to main content
Version: 10.3.1

Gold data normalization

This Business Process (BP) normalizes gold data values by doing the following:

  • Removes extra spaces.
  • Removes tabs and newlines.

Input​

Format: CSV file

Structure: has a column that contains tagged XML/HTML for normalization. The column name by default is tagged_text, it can be changed on the BP params step.

Gold data normalization

Implemented logic​

  1. Reads XML (HTML) declaration as a string.
  2. Removes all artifacts (blank spaces, tabs, new lines) from the string.
  3. Converts the string to a document format.
  4. Recursively goes through the document and fixes all data-value attributes (removes blank spaces, tabs, new lines).
  5. Converts the document to a string.
  6. Overwrites respective values in an input file.

Output​

Format: CSV file

Structure: same as input