Manage datasets

A dataset is a container for documents and all related meta information, such as:

• Labels
• Schema
• OCR settings
• Statistics, and so on

To create and manage different types of datasets, go to AutoML > Datasets. Control Tower updates the list every time you add a dataset.

In the Datasets list, you can find the following information about datasets.

Parameter	Description
Title	Dataset name specified during dataset creation. Each dataset name is a link to the associated dataset details page.
Type	Dataset type: tabular or document.
Volume	Number of rows for a tabular dataset or number of documents for a document dataset.
Author	Name or alias of a dataset author.
Created	Date and time (in the dd.mm.yyyy, hh:mm format) when the dataset was created.

The flow to manage a particular dataset depends on its type:

• Tabular: a CSV-based dataset used for classification models. Mind that a CSV file must contain the document content in the first column and gold values in the second one. Additional columns are passed as is.
• Document: a dataset containing TXT, TIFF, PDF, or any image files. The type can be used for classification and information extraction models.

Filter and search

You can search datasets in the list of available datasets or filter them based on their type.

• To search for a particular dataset by its title or author, use the Search field in the top-left corner.

• To filter datasets by their type, click the Type dropdown and select a needed dataset type: Document, Tabular, or both. The list is filtered accordingly.

  

  To turn off an active filter, click the Close icon next to Type.

You can sort datasets by their titles, types, author, or the creation date and time by clicking a column name. Sorting by volume is not available.

To view detailed information about a dataset, go to AutoML > Datasets and click the dataset's title in the list or open its Actions menu and select View details.

The dataset Overview window appears. You can use the breadcrumbs to navigate within Datasets.

Create dataset

To create a dataset, follow the steps below:

1. In the top-right corner of the Datasets page, click the plus (+) button and select Create new.

   

2. In the Dataset name field, specify a name for the dataset you are creating.

   Create a tabular dataset

   1. To create a tabular dataset, select Tabular. Add a description if needed.

      

   2. Click Create. A new tabular dataset is added to the dataset list, and the Overview window appears:

      

   Before you upload a dataset, the Rows data item shows 0, and Data source is empty.

   3. To upload a dataset file, click the plus (+) button in the top-right corner.

      

   4. In the file selection dialog, specify the path to a CSV file and click Open to start the upload.

   caution

   Before uploading, check that your CSV file contains the document content in the first column and gold values in the second one. Additional columns are passed as is.

   Once your dataset file is uploaded, you can see the exact number of columns and rows and the dataset statistics in the Overview window. The uploaded CSV file is available as Data source.

   

   Create a document dataset

   1. To create a document dataset, select Document. Add a description if needed.

      

   2. Click Create. A new document dataset appears in the dataset list, and the Documents tab is displayed.

   3. To upload required files, drag them to the Documents area, or click Browse or the + Documents button.

      

   4. Specify the path to your dataset files, and click Open in the window that follows. For a document dataset, you can upload TXT, TIFF, PDF, or any image files. Once your documents are uploaded, you can see them on the Documents tab.

      

   On the Details tab, in the Documents column, you can find the total number of uploaded documents.

   

Modify dataset

To override a dataset with new data, go to AutoML > Datasets, and click a dataset title.

• For a tabular dataset, in the Overview window, click the plus (+) button. In the window that follows, specify the path to a new CSV file and click Open to start the upload.

• For a document dataset, on the Documents tab, click the + Documents button. In the window that follows, specify the path to new files and click Open to start the upload. You can upload TXT, TIFF, PDF, or any image files.

Rename dataset

You can rename a dataset in one of the following ways:

• Go to AutoML > Datasets, select a dataset you want to rename, and click Actions > Rename.

  

• In the top-right corner of the Overview window, open the Actions menu and select the Rename option.

  

In the window that follows, in the Dataset name field, specify a new dataset name and click Save.

Copy dataset

Instead of creating a dataset from scratch, you can copy an existing one and modify it to your needs. To copy, proceed in one of the following ways:

• Go to AutoML > Datasets, select the dataset you want to copy, open the Actions menu, and click Copy.

  

  A new dataset appears in the Datasets list. The dataset name has the following format: original dataset name + copy + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

  Also, you see a confirmation message at the bottom of the screen. To go to a copied dataset, click View on the confirmation message or select it in the list of datasets.

  

• In the top-right corner of the Overview window, open the Actions menu and select Copy.

  

  You are automatically redirected to a copied dataset.

  

Once you complete the copy action, proceed to rename and modify the copy as you think appropriate.

Export dataset

To export a dataset as a bundle (a zip archive), find a dataset you want to export in the Datasets list, open its Actions menu, and select Export.

The resulting archive contains the following files:

• dataset.json comprises a full dataset representation, including schema, properties, tags, and so on.
• observations.csv contains observations or feature values (if any) in the CSV format.
• meta-info.json includes the technical information associated with the bundle, such as the creation date, author, and the Work.AI platform version.
• The files folder contains documents included in the dataset. The files directory is available only if you exported a document dataset.

Import dataset

To add a dataset to the list of available datasets, you can import it as a bundle similar to that created with the Export function. Each imported dataset must have a unique ID.

To import a dataset, in the top-right corner of the Datasets page, click the plus (+) button and select Import.

Above the Datasets list, you will see the import progress bar.

The dataset is imported, you see a confirmation message in the bottom-right corner of the screen, and the dataset appears in the list. Otherwise, you get a warning or an error message.

Publish dataset version

You can create dataset versions as immutable snapshots and use them to train and test models or to restore the current dataset state.

To publish a new version, open your dataset's Actions menu and select Publish.

When a new dataset version is published, a confirmation message appears in the bottom-right corner of the screen.

info

You cannot publish a dataset until you make any changes. A message appears warning that the dataset version you are trying to publish is identical to the previous one.

To view data for a new dataset version, in the top-right corner, click the drop-down box with available dataset versions and select the version.

Documents for published versions are shown in the read-only mode.

You can perform the following actions with a published dataset:

• View labeled documents. In the read-only mode, you cannot change anything.

  

• Restore data. To make any changes to a selected dataset version, click Enable editing.

  

  In the popup window, click Enable editing to confirm your choice.

  

  A new dataset version is created immediately, while the previous one remains intact and is available from the dropdown with available dataset versions.

  

Manage tabular datasets

Click a dataset title in the list of datasets or open its Actions menu and select View details.

For a tabular dataset, you can see the following meta information:

Parameter	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Tabular.
Parent version	Dataset version (if any) used to publish a dataset. Click the version to view the parent dataset.
Columns/Rows	Number of columns and rows in the dataset CSV file.
Data source	Data source CSV file (if any). You can download the file by clicking the Download link next to the CSV file.

The Dataset statistics section is displayed if you upload a dataset file. If you update the dataset, the statistics are updated accordingly. You can sort the table by columns.

Manage document datasets

Click a dataset title in the list of datasets and select the Details tab.

For a document dataset, you can see the following meta information:

Data item	Description
Created	Date and time (dd.mm.yyyy, hh:mm) when the dataset was created.
Author	Name or alias of a dataset author.
Description	Dataset description (if any).
Data type	Document.
Parent version	Dataset version (if any) used to publish a dataset. Click the version to view the parent dataset.
Documents	Number of documents in the dataset.

Select documents

To view all available documents in a dataset, use the navigation menu in the right-bottom corner.

The following information is available for each document:

Parameter	Description
Document	Document name.
Added	Date and time (in the dd.mm.yyyy, hh:mm format) when the document was added to the dataset.
Status	Document status: Unlabeled or Labeled.

The options for bulk actions are not active until you select at least one document. You can perform the following bulk actions with selected documents:

• Label
• Extract to dataset
• Delete

To select all documents (not only those visible on the page), select the checkbox next to the Document header.

To cancel the selection, deselect the checkbox next to the Document header.

Sort documents

You can sort documents by names, date of adding, and labeling status by clicking the appropriate column.

To filter documents by status, click Status and select Unlabeled or Labeled. The documents are filtered accordingly. To turn off an active filter, click Close next to it.

You can also search the list for a particular document by its name. In the Search field in the top-right corner, specify a document name. The results are displayed automatically.

Preview document

To preview a document's content, on the Documents tab, select a document in the list and click it. Alternatively, open the Actions menu and click Preview.

There are several options available on the Preview screen:

1. Select a document for future actions. When you return to the list of documents, the document stays selected.
2. Navigate between the documents using the Previous and Next options.
3. To delete a document, click the Actions button and select Remove.
4. To exit the preview mode and return to the list of documents, click the All documents button.

You can preview TXT, PDF, or any image files. If a document is not available for preview, for example, in the case of the GIF and TIFF formats, you can download it by clicking Download the file.

Extract documents to dataset

You can create a new dataset using documents from an existing dataset:

1. On the Documents tab, select documents to add to a new dataset.

2. In the bulk actions menu, select the Extract to dataset option.

   

   You are automatically redirected to a new dataset > the Documents tab. A confirmation message appears in the bottom-right corner of the screen.

   

The dataset is of the same type and contains the same labels. All labeled documents are copied as labeled. The dataset name has the following format: Extracted from + original dataset name + date and time. Proceed to renaming and modifying the dataset as you think appropriate.

Configure OCR settings

On the Documents tab, you can configure OCR settings for processing your documents. To open the configuration window, click the OCR settings button on the right.

To hide the OCR settings, click the Hide button.

In the OCR providers drop-down list, you can select a tool for recognizing documents in your dataset. There are three OCR providers to choose from:

• ABBY
• Google
• Azure

info

For Azure and Google, you can specify the following settings only:

• Enable native PDF processing
• Convert documents to TIFF

For ABBYY, you can configure all the settings listed below.

There are two modes you can choose from when configuring OCR settings:

Basic

The Basic mode allows you to configure the settings using only user interface controls without coding.

Available OCR settings are as follows:

• Native PDF processing: enable native PDF processing and process all native PDFs without OCR. The feature extracts only searchable text and reduces your OCR license usage.

• Languages: Add language tags to specify the languages that are likely to be found in your documents. You can add a maximum of six languages. Subsequently, the engine automatically chooses one of them as the most suitable for reading each particular document.

• Detect language: select if you want the engine to read texts word by word, applying the most suitable language from the Languages box.

• Image correction options:

  • Change DPI: set the DPI resolution range of the recognition output. The available values are from 50 to 600.

  • Correct orientation: select if you want the engine to adjust the orientation of the recognized document.

  • Invert image: invert the colors of the recognized document.

  • Discard color image: replace any colors of an original document with the shades of grey in the recognition output.

  • Enable low-resolution mode: select to improve the recognition output from low-quality originals.

  • Enhance contrast: select to improve the contrast of the recognized document.

  • Detect text in images: select if you want the engine to recognize any text in images (for example, logos) inside the document.

  • Convert document to TIFF: convert documents to the TIFF format before recognition.

Advanced

The Advanced mode allows you to set the same OCR settings you configured in the Basic mode but in the JSON representation.

note

Any changes made in the Basic mode are reflected on the Advanced tab and vice versa.

Click Save to apply the changes and confirm the action in the confirmation dialog. To cancel any changes you made, select Revert changes. If you make any changes and try to leave the page, a warning message about unsaved changes appears.

Set up labels

If labels are not configured for your dataset, a yellow icon is displayed next to the Labels tab title. To add labels, go to the Labels tab and set them up in one of the following ways:

• Use Builder.
• Edit the JSON representation.
• Generate labels from a template.

Configure labels in Builder

You can set up labels using the no-code approach:

1. Go to the Labels tab > Builder.

2. Drag the needed labels from the left panel to the right one.

   

   Once a label is moved to the right panel, its detailed setup opens in a separate window. Change any settings as you need or leave the default setup.

   

   For Text field components, go to the Data tab, open the Answer Type menu, and specify the field type for your model. For other controls, you can leave the default setup.

   

3. Click Save. The labels you added become visible on the right.

As you move the pointer over a label, a menu opens, and you can select an action:

• Edit allows editing the form component parameters in Builder.

• Edit JSON enables editing the form component parameters in JSON representation. The Preview option allows viewing edits as they will look in the form.

  

• Move allows you to move the form component within the form.

• Copy enables copying the form component. Click the Copy button, move the pointer over the form component below which you want to insert the copied one, and click Paste below.

  

  The form component appears in the form. In the example, the Date/Time form component was copied and pasted below the Radio component.

  

• Remove allows deleting the form component from the form.

caution

Changing label names (Edit > Display tab > Label) can result in loss of labeled data as the names are used during the labeling process.

Edit JSON representation

You can configure labels directly in the code:

1. Go to the Labels tab > Code.

   

2. Make changes in the JSON code.

   • You can copy and paste the code.
   • You can undo the unsaved changes by clicking Revert changes.

3. Click Save.

   

   You cannot save invalid JSON. If there are any errors, you see a warning message. Fix the issues and click Save again.

   

If you try switching to Builder or any other tab with unsaved changes, a warning dialog appears as shown below. Click Ok to close the dialog and make sure to save or revert changes.

Generate labels from template

For detailed instructions on generating labels from a template, read Create and manage templates | Use templates to generate labels in datasets.

Label documents

To start the labeling flow, go to the Documents tab and follow the steps below:

1. Select the documents you want to label and click the Label button.

   

   info

   If a previously labeled document is passed to labeling, the previously labeled data is restored.

2. Choose a labeling method and proceed as described in respective instructions:

   • Label manually if you want to process the documents yourself.

   • Label with AI if you want a model to label the documents for you automatically.

   

caution

Up to 10 users in parallel can label documents in a dataset using the default memory configuration for dataset management and labeling services.

Manual labeling

After you choose Label manually, click the Start labelling button. The action redirects you to the page with the document you need to label. For labeling instructions, see the documentation.

To save your work after you finish labeling, click Finish. If you perform bulk labeling (more than one document), proceed to the next document in the queue by clicking Next.

Labeled data is saved to the dataset. On the Documents tab, the Status for the processed documents is changed to Labeled .

Labeling with AI

After you choose Label with AI, do as follows:

1. Select a model provider for automatic labeling.

   

   caution

   Using OpenAI or MistralAI as the model provider is only possible if you enabled the auto-labeling feature. For instructions, see the Enable auto labeling with OpenAI or MistralAI guide.

2. Depending on the chosen model provider, proceed in one of the following ways:

   • If you opt for WorkFusion to assist you with labeling, choose a model from the Model drop-down list.

     

   • If you choose OpenAI or MistralAI to assist you with labeling, select a model and select a document type. To add a custom document type, type it in the Document Type field.

     

   caution

   When using OpenAI or MistralAI models, you are charged additionally by the mentioned providers according to your pricing plan. No additional charges are applied when you choose a WorkFusion model.

3. Click Start labelling to launch the auto labeling flow. The model starts labeling the documents for you.

On the Documents tab, all documents that are being processed by the model have the Labeling with AI in progress status. The successfully processed ones have the Labeled with AI status. Labeled data is saved to the dataset.

Once the model finishes labeling all selected documents, a corresponding warning appears in the bottom right corner.

If you click View results inside the warning, a page opens where you can review the results of the model-assisted labeling and make changes if necessary. Any validation errors are visible in the UI form.

If you perform bulk labeling, proceed to the next document in the queue by clicking Next. To save the labeling results, click Finish.

If anything goes wrong during labeling, the corresponding document on the Documents list gets the Labeling with AI failed status. Also, you see an appropriate warning in the top bottom corner.

Delete document

You can delete documents in the following ways:

• Go to the Documents tab, select a document, open its Actions menu, and click Delete.

  

• Enter the preview mode, click the Actions button, and select Delete.

  

• To perform a bulk removal, on the Documents tab, select the documents you want to remove and click the Delete button.

  

  The selected documents are deleted, and you see a confirmation message in the bottom-right corner of the screen.

  

Delete dataset

To remove a dataset, perform the following actions:

1. Find the dataset you want to delete in the Datasets list, open its Actions menu, and click Delete.

   

   You can also select the Delete option when you view your dataset.

   

2. In the window that follows, click the Yes, delete button to confirm the deletion.

   

   The selected dataset is deleted, and you see a confirmation message in the bottom-right corner of the screen.