Edit

Manage datasets

A dataset is a container for documents and all related meta information, such as:

• Labels
• Schema
• OCR settings
• Statistics, and so on

To create and manage different types of datasets, go to AutoML > Datasets. Control Tower updates the list every time you add a dataset.

In the Datasets list, you can find the following information about datasets.

Parameter	Description
Title	Dataset name specified during dataset creation. Each dataset name is a link to the associated dataset details page.
Type	Dataset type: tabular or document.
Volume	Number of rows for a tabular dataset or number of documents for a document dataset.
Author	Name or alias of a dataset author.
Created	Date and time (in the dd.mm.yyyy, hh:mm format) when the dataset was created.

The flow to manage a particular dataset depends on its type:

• Tabular: a CSV-based dataset used for classification models. Mind that a CSV file must contain the document content in the first column and gold values in the second one. Additional columns are passed as is.
• Document: a dataset containing TXT, TIFF, PDF, or any image files. The type can be used for classification and information extraction models.

Filter and search

You can filter the list of datasets based on their type. Click the Type filter button and specify the dataset type: Document or Tabular. The list is filtered accordingly. To turn off an active filter, click it again.

You can sort datasets by their titles, types, author, or the creation date and time by clicking a column name. Sorting by volume is not available.

You can also search for a particular dataset by its title or author using the Search box in the top-right corner.

To view detailed information about a dataset, go to AutoML > Datasets and click the dataset's title in the list or open its Actions menu and select View details.

The dataset Overview window appears. You can use the breadcrumbs to navigate within Datasets.

Create dataset

To create a dataset, follow the steps below:

1. In the top-right corner of the Datasets page, click the plus (+) button and select Create new.

   

2. In the Dataset name field, specify a name for the dataset you are creating.

   Create a tabular dataset

   1. To create a tabular dataset, select Tabular. Add a description if needed.

      

   2. Click Create. A new tabular dataset is added to the dataset list, and the Overview window appears:

      

   Before you upload a dataset, the Rows data item shows 0, and Data source is empty.

   3. To upload a dataset file, click the plus (+) button in the top-right corner.

      

   4. In the file selection dialog, specify the path to a CSV file and click Open to start the upload.

   caution

   Before uploading, check that your CSV file contains the document content in the first column and gold values in the second one. Additional columns are passed as is.

   Once your dataset file is uploaded, you can see the exact number of columns and rows and the dataset statistics in the Overview window. The uploaded CSV file is available as Data source.

   

   Create a document dataset

   1. To create a document dataset, select Document. Add a description if needed.

      

   2. Click Create. A new document dataset appears in the dataset list, and the Overview tab is displayed:

      

   3. Navigate to the Documents tab. Before you upload any documents, the page is empty.

   4. To upload required files, drag them to the Documents area, or click Browse or the + Documents button.

      

   5. Specify the path to your dataset files, and click Open in the window that follows. For a document dataset, you can upload TXT, TIFF, PDF, or any image files. Once your documents are uploaded, you can see them on the Documents tab.

      

   On the Overview tab, in the Documents column, you can find the total number of uploaded documents.

   

Modify dataset

To override a dataset with new data, go to AutoML > Datasets, and click the title of a dataset to update.

• For a tabular dataset, in the Overview window, click the plus (+) button. In the window that follows, specify the path to a new CSV file and click Open to start the upload.

  

• For a document dataset, on the Documents tab, click the + Documents button. In the window that follows, specify the path to new files and click Open to start the upload. You can upload TXT, TIFF, PDF, or any image files.

  

Rename dataset

You can rename a dataset in one of the following ways:

• Go to AutoML > Datasets, select a dataset you want to rename, and click Actions > Rename.

  

• In the top right corner of the Overview window, open the Actions menu and select the Rename option.

  

In the window that follows, in the Dataset name field, specify a new dataset name and click Save.

Copy dataset

Instead of creating a dataset from scratch, you can copy an existing one and modify it to your needs. To copy, proceed in one of the following ways:

• Go to AutoML > Datasets, select the dataset you want to copy, open the Actions menu, and click Copy.

  

  A new dataset appears in the Datasets list. The dataset name has the following format: <original dataset name> + Copy + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

  Also, you see a confirmation message in the bottom-right corner of the screen. To go to a copied dataset, click View on the confirmation message or select it in the list of datasets.

  

• In the top-right corner of the Overview window, open the Actions menu and select Copy.

  

  You are automatically redirected to a copied dataset.

  

Once you complete the copy action, proceed to rename and modify the copy as you think appropriate.

Export dataset

To export a dataset as a bundle (a zip archive), find a dataset you want to export in the Datasets list, open its Actions menu, and select Export.

The resulting archive contains the following files:

• dataset.json comprises a full dataset representation, including schema, properties, tags, and so on.
• observations.csv contains observations or feature values (if any) in the CSV format.
• meta-info.json includes the technical information associated with the bundle, such as the creation date, author, and the WorkFusion platform version.
• The files folder contains documents included in the dataset. The files directory is available only if you exported a document dataset.

Import dataset

To add a dataset to the list of available datasets, you can import it as a bundle similar to that created with the Export function. Each imported dataset must have a unique ID.

To import a dataset, in the top-right corner of the Datasets list page, click the plus (+) button and select Import.

Above the Datasets list, you will see the import progress bar.

The dataset is imported, you see a confirmation message in the bottom-right corner of the screen, and the dataset appears in the list. Otherwise, you get a warning or an error message.

Manage tabular datasets

Click a dataset title in the list of datasets or open its Actions menu and select View details.

For a tabular dataset, you can see the following meta information:

Parameter	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Tabular.
Columns	Number of columns in the dataset CSV file.
Rows	Number of rows in the dataset CSV file.
Data source	Data source CSV file (if any). You can download the file by clicking the Download link next to the CSV file.

The Dataset statistics section is displayed if you upload a dataset file. If you update the dataset, the statistics are updated accordingly. You can sort the table by columns.

Manage document datasets

Click a dataset title in the list of datasets or open its Actions menu and select View details.

For a document dataset, you can see the following meta information:

Data item	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Document.
Documents	Number of documents in the dataset.

Select documents

To view all available documents in a dataset, use the scroll bar on the right.

The following information on each document is available:

Parameter	Description
Document	Document name.
Added	Date and time (in the dd.mm.yyyy, hh:mm format) when the document was added to the dataset.
Status	Document status: Unlabeled or Labeled.

You can perform the following bulk actions with selected documents:

• Label
• Extract to dataset
• Remove

The option for bulk actions is not active until you select at least one document.

To select all documents (not only those visible on the page), select the checkbox next to the Document column title.

The number of selected documents is indicated next to the Status filter. To cancel the selection, deselect the checkbox next to the Document column title.

Sort documents

You can sort documents by names, date of adding, labels, and status by clicking the appropriate column.

To filter documents by status, click Status and select Unlabeled or Labeled. The documents are filtered accordingly. To turn off an active filter, click it again.

You can also search the list for a particular document by its name. In the Search box in the top right corner, specify a document name. The results are displayed automatically.

Preview document

To preview a document's contents, on the Documents tab, select a document in the list and click it.

There are several options available on the Preview screen:

1. Select a document for future actions. When you return to the list of documents, the document stays selected.
2. Navigate between the documents using the Previous and Next options.
3. To delete a document, click the Actions button and select Remove.
4. To exit the preview mode and return to the list of documents, click the All documents button.

You can preview TXT, PDF, or any image files. If a document is not available for preview, for example, in the case of the GIF and TIFF formats, you can download it by clicking Download the file.

Extract documents to dataset

You can create a new dataset using documents from the existing dataset:

1. On the Documents tab, select documents to add to a new dataset.

2. Click the tick button and, in the bulk actions menu, select the Extract to dataset option.

   

   You are automatically redirected to a new dataset. A confirmation message appears in the bottom-right corner of the screen.

   

The dataset is of the same type and contains the same labels. All labeled documents are copied as labeled. The dataset name has the following format: Extracted from + <original dataset name> + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

The extracted documents are available on the Documents tab.

Configure OCR settings

On the Documents tab, you can configure OCR settings for processing your documents. To expand the configuration window, click the OCR settings button on the right.

To hide the OCR settings, click the Hide button:

You can configure the following settings:

• OCR providers: select a tool for recognizing documents in your dataset. There are three OCR providers to choose from:

  • ABBY
  • GOOGLE
  • AZURE

  

• Native PDF processing: select the checkbox to enable native PDF processing and process all native PDFs without OCR. The feature extracts only searchable text and reduces your OCR license usage.

• OCR parameters: add or edit OCR parameters in the JSON representation.

Click Save to apply the changes.

Set up labels

If labels are not configured for your dataset, a yellow icon is displayed next to the Labels tab title.

To add labels, go to the Labels tab. You can configure labels in two ways:

• Use Builder.
• Edit a JSON representation.

Configure labels in Builder

You can set up labels using the no-code approach:

1. Go to the Labels tab > Builder.

2. Drag the needed labels from the left panel to the right one.

   

   Once a label is moved to the right panel, its detailed setup opens in a separate window.

   

   For Text field components, go to the Data tab, open the Answer Type menu, and specify the field type for your model.

   

   For other controls, you can leave the default setup.

3. Click Save. The labels you added are visible on the right.

To modify any label settings, move the pointer over a label and select one of the available actions:

• Edit: Edit form component parameters in Builder.

• Edit JSON: Edit form component parameters in a JSON representation. The Preview option allows viewing edits as they will look in the form.

  

• Move: Move a form component within the form.

• Copy: Copy a form component. Once you click the Copy button, move a pointer over a form component below which you want to insert a copied one, and click Paste below.

  

  The form component appears in the form. In the example, the Date/Time form component was copied and pasted below the Radio component.

  

• Remove: Delete a form component from the form.

caution

Mind that changing label names (Edit > Display tab > Label) can result in loss of labeled data as the names are used for labeling.

Edit JSON representation

You can configure labels directly in the code:

1. Go to the Labels tab > Code.

   

2. Enter your changes in the JSON.

   note

   • You can copy and paste the code.
   • You can revert unsaved changes by clicking Revert changes.

3. Click Save once finished.

   

   You cannot save an invalid JSON. If there are any errors, you see a warning message.

   

   Fix the issues and click Save again.

You cannot switch to Builder or any other tab if you have unsaved changes. In the dialog window that follows, click Ok.

Then, click Revert changes to restore the configuration or Save to save the changes.

Label documents

To label your dataset documents, go to the Documents tab and perform the following actions:

1. Select the documents you want to label, click the tick button next to the Status filter, and choose the Label option.

   

   note

   The tick button for bulk actions is inactive until you select at least one document.

   If you use the checkbox to select all documents (next to the Document column title), only those documents that are visible on the page are selected. The number of selected documents out of their total quantity is indicated next to the Status filter. Use the scroll bar to see all available documents and select more.

   

   important

   If a previously labeled document is passed to labeling, the previously labeled data is restored.

2. On the page that follows, label your document. For more instructions on labeling, see Start tagging.

   

3. To save, click Finish. If you perform bulk labeling, proceed to the next document in a queue by clicking Next. To complete the labeling procedure and save the progress for your labeled documents, select Finish.

   

   Labeled data is stored in the dataset. On the Documents tab, the Status for the processed documents is changed to Labeled .

   

caution

Up to 10 users in parallel can label documents in a dataset using the default memory configuration for dataset management and labeling services.

Remove document

You can remove documents in the following ways:

• Go to the Documents tab, select a document, open its Actions menu, and click Remove.

  

• Enter the preview mode, click the Actions button, and select Remove.

  

• To perform a bulk removal, on the Documents tab, select the documents you want to remove, click the tick button next to the Status filter, and select the Remove option.

  

The selected documents are removed, and you see a confirmation message in the bottom-right corner of the screen.

Remove dataset

To delete a dataset, perform the following actions:

1. Find the dataset you want to remove in the Datasets list, open its Actions menu, and click Remove.

   

2. In the window that follows, click the Yes, delete button to confirm the deletion.

   

   The selected dataset is removed, and you see a confirmation message in the bottom-right corner of the screen.