Normalize data values

overview

Task: clean up or format data values in gold or extracted values.

Jupiter Notebook: GoldDataNormalization

Input:

• Tagged data saved to a Data Store
• A CSV file with tagged text

To normalize data values, follow the instruction below:

1. Launch all dependencies and prerequisites.

   

2. Upload data by any available method.

3. Import normalizing functions.

   

4. Open the widget for normalization, specify all data in the drop-downs boxes as indicated below:

   

   • Select a dataframe: enter the name of the dataframe that contains the dataset to normalize

   • Document field: specify the column in the dataframe that contains the data (tagged text) to normalize

   • Output field: add the name of the new column that contains normalized data (tagged text).

5. Click Analyze.

6. A widget to normalize data is opened with available fields and their data-value samples. Perform required normalization.

   Available normalization options are as follows:

   • Rename a field or change the name partially

   • Set a data value.

   • Format a data value.

   • Replace a data value.

   • Remove a field.

   • Bring the data value to a lower case.

   • Replace the data value with regexp.

   There are short instructions for each function and fields to input data.

   

7. To add multiple operations, click +.

   

   Sample result:

   

8. When data for normalization is provided, click Start processing.

   

9. Save the output to the dataframe and provide the path and file name to save the result locally.