Manage datasets

A dataset is a container for documents and all related meta information, such as:

• Labels
• Schema
• OCR settings
• Statistics, and so on

To create and manage different types of datasets, go to AutoML > Datasets. Control Tower updates the list every time you add a dataset.

In the Datasets list, you can find the following information about datasets.

Parameter	Description
Title	Dataset name specified during dataset creation. Each dataset name is a link to the associated dataset details page.
Type	Dataset type: tabular or document.
Volume	Number of rows for a tabular dataset or number of documents for a document dataset.
Author	Name or alias of a dataset author.
Created	Date and time (in the dd.mm.yyyy, hh:mm format) when the dataset was created.

The flow to manage a particular dataset depends on its type:

• Tabular: a CSV-based dataset used for classification models. Mind that a CSV file must contain the document content in the first column and gold values in the second one. Additional columns are passed as is.
• Document: a dataset containing TXT, TIFF, PDF, or any image files. The type can be used for classification and information extraction models.

Filter and search

You can search datasets in the list of available datasets or filter them based on their type.

• To search for a particular dataset by its title or author, use the Search field in the top-left corner.

• To filter datasets by their type, click the Type dropdown and select a needed dataset type: Document, Tabular, or both. The list is filtered accordingly.

  

  To turn off an active filter, click the Close icon next to Type.

You can sort datasets by their titles, types, author, or the creation date and time by clicking a column name. Sorting by volume is not available.

To view detailed information about a dataset, go to AutoML > Datasets and click the dataset's title in the list or open its Actions menu and select View details.

The dataset Overview window appears. You can use the breadcrumbs to navigate within Datasets.

Create dataset

To create a dataset, follow the steps below:

1. In the top-right corner of the Datasets page, click the plus (+) button and select Create new.

   

2. In the Dataset name field, specify a name for the dataset you are creating.

   Create a tabular dataset

   1. To create a tabular dataset, select Tabular. Add a description if needed.

      

   2. Click Create. A new tabular dataset is added to the dataset list, and the Overview window appears:

      

   Before you upload a dataset, the Rows data item shows 0, and Data source is empty.

   3. To upload a dataset file, click the plus (+) button in the top-right corner.

      

   4. In the file selection dialog, specify the path to a CSV file and click Open to start the upload.

   caution

   Before uploading, check that your CSV file contains the document content in the first column and gold values in the second one. Additional columns are passed as is.

   Once your dataset file is uploaded, you can see the exact number of columns and rows and the dataset statistics in the Overview window. The uploaded CSV file is available as Data source.

   

   Create a document dataset

   1. To create a document dataset, select Document. Add a description if needed.

      

   2. Click Create. A new document dataset appears in the dataset list, and the Documents tab is displayed.

   3. To upload required files, drag them to the Documents area, or click Browse or the + Documents button.

      

   4. Specify the path to your dataset files, and click Open in the window that follows. For a document dataset, you can upload TXT, TIFF, PDF, or any image files. Once your documents are uploaded, you can see them on the Documents tab.

      

   On the Details tab, in the Documents column, you can find the total number of uploaded documents.

   

Modify dataset

To override a dataset with new data, go to AutoML > Datasets, and click the title of a dataset to update.

• For a tabular dataset, in the Overview window, click the plus (+) button. In the window that follows, specify the path to a new CSV file and click Open to start the upload.

• For a document dataset, on the Documents tab, click the + Documents button. In the window that follows, specify the path to new files and click Open to start the upload. You can upload TXT, TIFF, PDF, or any image files.

Rename dataset

You can rename a dataset in one of the following ways:

• Go to AutoML > Datasets, select a dataset you want to rename, and click Actions > Rename.

  

• In the top-right corner of the Overview window, open the Actions menu and select the Rename option.

  

In the window that follows, in the Dataset name field, specify a new dataset name and click Save.

Copy dataset

Instead of creating a dataset from scratch, you can copy an existing one and modify it to your needs. To copy, proceed in one of the following ways:

• Go to AutoML > Datasets, select the dataset you want to copy, open the Actions menu, and click Copy.

  

  A new dataset appears in the Datasets list. The dataset name has the following format: <original dataset name> + Copy + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

  Also, you see a confirmation message at the bottom of the screen. To go to a copied dataset, click View on the confirmation message or select it in the list of datasets.

  

• In the top-right corner of the Overview window, open the Actions menu and select Copy.

  

  You are automatically redirected to a copied dataset.

  

Once you complete the copy action, proceed to rename and modify the copy as you think appropriate.

Export dataset

To export a dataset as a bundle (a zip archive), find a dataset you want to export in the Datasets list, open its Actions menu, and select Export.

The resulting archive contains the following files:

• dataset.json comprises a full dataset representation, including schema, properties, tags, and so on.
• observations.csv contains observations or feature values (if any) in the CSV format.
• meta-info.json includes the technical information associated with the bundle, such as the creation date, author, and the Work.AI platform version.
• The files folder contains documents included in the dataset. The files directory is available only if you exported a document dataset.

Import dataset

To add a dataset to the list of available datasets, you can import it as a bundle similar to that created with the Export function. Each imported dataset must have a unique ID.

To import a dataset, in the top-right corner of the Datasets page, click the plus (+) button and select Import.

Above the Datasets list, you will see the import progress bar.

The dataset is imported, you see a confirmation message in the bottom-right corner of the screen, and the dataset appears in the list. Otherwise, you get a warning or an error message.

Publish dataset

You can create dataset versions as immutable snapshots and use them to train and test models or to restore the current dataset state.

To publish a new version, open your dataset's Actions menu and select Publish.

When a new dataset version is published, a confirmation message appears in the bottom-right corner of the screen.

info

Mind that you cannot publish a dataset until you make any changes. A message appears warning that the dataset version you are trying to publish is identical to the previous one.

To view data for a new dataset version, in the top-right corner, click the dropdown with available dataset versions and select the version.

Documents for published versions are shown in the read-only mode.

You can perform the following actions with a published dataset:

• View labeled documents. In the read-only mode, you cannot change anything.

  

• Restore data. To make any changes to a selected dataset version, click Enable editing.

  

  In the popup window, click Enable editing to confirm your choice.

  

  A new dataset version is created immediately, while the previous one remains intact and is available from the dropdown with available dataset versions.

  

Manage tabular datasets

Click a dataset title in the list of datasets or open its Actions menu and select View details.

For a tabular dataset, you can see the following meta information:

Parameter	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Tabular.
Parent version	Dataset version (if any) used to publish a dataset. Click the version to view the parent dataset.
Columns/Rows	Number of columns and rows in the dataset CSV file.
Data source	Data source CSV file (if any). You can download the file by clicking the Download link next to the CSV file.

The Dataset statistics section is displayed if you upload a dataset file. If you update the dataset, the statistics are updated accordingly. You can sort the table by columns.

Manage document datasets

Click a dataset title in the list of datasets and select the Details tab.

For a document dataset, you can see the following meta information:

Data item	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Document.
Parent version	Dataset version (if any) used to publish a dataset. Click the version to view the parent dataset.
Documents	Number of documents in the dataset.

Select documents

To view all available documents in a dataset, use the navigation menu in the right-bottom corner.

The following information is available for each document:

Parameter	Description
Document	Document name.
Added	Date and time (in the dd.mm.yyyy, hh:mm format) when the document was added to the dataset.
Status	Document status: Unlabeled or Labeled.

The option for bulk actions is not active until you select at least one document. You can perform the following bulk actions with selected documents:

• Label
• Extract to dataset
• Delete

To select all documents (not only those visible on the page), select the checkbox next to the Document column title.

The number of selected documents is indicated next to the Status filter. To cancel the selection, deselect the checkbox next to the Document column title.

Sort documents

You can sort documents by names, date of adding, labels, and status by clicking the appropriate column.

To filter documents by status, click Status and select Unlabeled or Labeled. The documents are filtered accordingly. To turn off an active filter, click Close next to it.

You can also search the list for a particular document by its name. In the Search field in the top-right corner, specify a document name. The results are displayed automatically.

Preview document

To preview a document's contents, on the Documents tab, select a document in the list and click it.

There are several options available on the Preview screen:

1. Select a document for future actions. When you return to the list of documents, the document stays selected.
2. Navigate between the documents using the Previous and Next options.
3. To delete a document, click the Actions button and select Remove.
4. To exit the preview mode and return to the list of documents, click the All documents button.

You can preview TXT, PDF, or any image files. If a document is not available for preview, for example, in the case of the GIF and TIFF formats, you can download it by clicking Download the file.

Extract documents to dataset

You can create a new dataset using documents from the existing dataset:

1. On the Documents tab, select documents to add to a new dataset.

2. Click the tick button and, in the bulk actions menu, select the Extract to dataset option.

   

   You are automatically redirected to a new dataset > the Documents tab. A confirmation message appears in the bottom-right corner of the screen.

   

The dataset is of the same type and contains the same labels. All labeled documents are copied as labeled. The dataset name has the following format: Extracted from + <original dataset name> + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

Configure OCR settings

On the Documents tab, you can configure OCR settings for processing your documents. To open the configuration window, click the OCR settings button on the right.

To hide the OCR settings, click the Hide button.

In the OCR providers drop-down list, you can select a tool for recognizing documents in your dataset. There are three OCR providers to choose from:

• ABBY
• Google
• Azure

info

For Azure and Google, you can specify the following settings only:

• Enable native PDF processing
• Convert documents to TIFF

For ABBYY, you can configure all the settings enumerated below.

There are two modes you can choose from when configuring OCR settings:

Basic

The Basic mode allows you to perform configuration using UI controls.

Available OCR settings are as follows:

• Native PDF processing: enable native PDF processing and process all native PDFs without OCR. The feature extracts only searchable text and reduces your OCR license usage.

• Languages: Add language tags to specify the languages that are likely to be found in your documents. You can add a maximum of six languages. Subsequently, the engine automatically chooses one of them as the most suitable for reading each particular document.

• Detect language: select if you want the engine to read texts word by word, applying the most suitable language from the Languages box.

• Image correction options:

  • Change DPI: set the DPI resolution range of the recognition output. The available values are from 50 to 600.

  • Correct orientation: select if you want the engine to adjust the orientation of the recognized document.

  • Invert image: invert the colors of the recognized document.

  • Discard color image: replace any colors of an original document with the shades of grey in the recognition output.

  • Enable low-resolution mode: select to improve the recognition output from low-quality originals.

  • Enhance contrast: select to improve the contrast of the recognized document.

  • Detect text in images: select if you want the engine to recognize any text in images (for example, logos) inside the document.

  • Convert document to TIFF: convert documents to the TIFF format before recognition.

Advanced

The Advanced mode allows you to set the same OCR settings you configured in the Basic mode but in the JSON representation.

note

Any changes made in the Basic mode are reflected on the Advanced tab and vice versa.

Click Save to apply the changes. To cancel any changes you made, select Revert changes. If you make any changes and try to leave the page, a warning message about unsaved changes appears.

Set up labels

If labels are not configured for your dataset, a yellow icon is displayed next to the Labels tab title.

To add labels, go to the Labels tab. You can configure labels in two ways:

• Use Builder.
• Edit a JSON representation.

Configure labels in Builder

You can set up labels using the no-code approach:

1. Go to the Labels tab > Builder.

2. Drag the needed labels from the left panel to the right one.

   

   Once a label is moved to the right panel, its detailed setup opens in a separate window.

   

   For Text field components, go to the Data tab, open the Answer Type menu, and specify the field type for your model.

   

   For other controls, you can leave the default setup.

3. Click Save. The labels you added are visible on the right.

To modify any label settings, move the pointer over a label and select one of the available actions:

• Edit: Edit form component parameters in Builder.

• Edit JSON: Edit form component parameters in a JSON representation. The Preview option allows viewing edits as they will look in the form.

  

• Move: Move a form component within the form.

• Copy: Copy a form component. Once you click the Copy button, move a pointer over a form component below which you want to insert a copied one, and click Paste below.

  

  The form component appears in the form. In the example, the Date/Time form component was copied and pasted below the Radio component.

  

• Remove: Delete a form component from the form.

caution

Mind that changing label names (Edit > Display tab > Label) can result in loss of labeled data as the names are used for labeling.

Edit JSON representation

You can configure labels directly in the code:

1. Go to the Labels tab > Code.

   

2. Enter your changes in the JSON.

   note

   • You can copy and paste the code.
   • You can revert unsaved changes by clicking Revert changes.

3. Click Save once finished.

   

   You cannot save an invalid JSON. If there are any errors, you see a warning message.

   

   Fix the issues and click Save again.

You cannot switch to Builder or any other tab if you have unsaved changes. In the dialog window that follows, click Ok.

Then, click Revert changes to restore the configuration or Save to save the changes.

Label documents

To label your dataset documents, go to the Documents tab and perform the following actions:

1. Select the documents you want to label and click the Label button.

   

   info

   If a previously labeled document is passed to labeling, the previously labeled data is restored.

2. On the page that follows, label your document. For more instructions on labeling, see the documentation.

   

3. To save, click Finish. If you perform bulk labeling, proceed to the next document in a queue by clicking Next. To complete the labeling procedure and save the progress for your labeled documents, select Finish.

   

   Labeled data is stored in the dataset. On the Documents tab, the Status for the processed documents is changed to Labeled .

   

caution

Up to 10 users in parallel can label documents in a dataset using the default memory configuration for dataset management and labeling services.

Delete document

You can delete documents in the following ways:

• Go to the Documents tab, select a document, open its Actions menu, and click Delete.

  

• Enter the preview mode, click the Actions button, and select Delete.

• To perform a bulk removal, on the Documents tab, select the documents you want to remove and click the Delete button.

  

  The selected documents are deleted, and you see a confirmation message in the bottom-right corner of the screen.

  

Delete dataset

To remove a dataset, perform the following actions:

1. Find the dataset you want to delete in the Datasets list, open its Actions menu, and click Delete.

   

   You can also select the Delete option when you view your dataset.

   

2. In the window that follows, click the Yes, delete button to confirm the deletion.

   

   The selected dataset is deleted, and you see a confirmation message in the bottom-right corner of the screen.