Skip to main content
Version: 10.2.8

Gold data normalization

This Business Process normalizes gold data values by doing the following:

  • Removes extra spaces.
  • Removes tabs and newlines.

Input​

Format: CSV file

Structure: Has column that contains tagged XML/HTML for normalization. Column name by default is tagged_text, it can be changed on the BP params step.

Gold data normalization

Implemented logic​

  1. Reads XML (HTML) declaration as a string.
  2. Removes all artifacts (blank spaces, tabs, new lines) from the string.
  3. Converts the string to a Document format.
  4. Recursively goes through the Document and fixes all data-value attributes (removes blank spaces, tabs, new lines).
  5. Converts the Document to a string.
  6. Overwrites respective values in an input file.

Output​

Format: CSV file.

Structure: same as input