Skip to main content
Version: 10.2.9

Input documents analysis

Goal: To understand the business logic and documents.

Input: Statement of Work (project contract with client where final success criteria must be defined), list and description of fields and classes, original documents, any other description or documentation provided by client.

Output: Logic defined for documents: formats, layouts, special business logic, fields or classes and their meaning, corner cases.

Steps:

  1. Understand the business logic from the documents and conversations.
  2. Study documents intended to form ML training set, analyze quality and representativeness of different document structures and distribution.
  3. Study fields logic, consider answer types and rare fields.
  4. Finalize fields output format.
  5. Consider Manual Task design and logic.

Most of the projects are delivered on the customer's premises, so DA needs to go to the customer. Steps at this stage can be divided into two parts:

  1. What you can do before you arrive.
  2. What should be done once on-site.

Before you begin​

Make sure all the credentials are received:

  • Access to folder with original documents.
  • S3 with full access at least on one bucket to upload documents into as well as standard AutoML model buckets to access results and configuration.
  • Tested Control Tower and Workspace instances with full access.

Make sure the required software is installed:

  • S3 Browser
  • LibreOffice, OpenOffice, Ron's Editor, or any other viewer of large CSV files
  • Chrome 63+
  • Notepad++
  • Microsoft Outlook
  • Webex, GTM, Skype for business or any other communication tool

Before arrival​

First, the Data Analyst should study the SOW carefully and make sure the whole Business Process is clear. It's necessary to figure out the project scope and documents' workflow. Find out whether there any specific details depending on the processing period during the year (for example, in the fourth quarter, does the flow contain the documents of some specific issuers with unique structure, or are the document structures different from period to period).

Here is the list of questions to be answered:

  1. What type is it—Information Extraction or Classification?
  2. What types of the documents do you have? How many of them are there?
  3. What document formats do you have?
  4. How many pages are there in one document? What pages are needed to be processed?
  5. How many fields are to be extracted? What are they?
  6. How many classes are to be formed? What are they?
  7. How many models are required to process the document flow?

Consider additional questions that may arise and ask them in advance.

On arrival​

If you have some documents' samples provided by customer, try to figure out from them as much as possible about documents' logic, structures, diversity and appearance, categories, fields' shapes and context before arrival.

In most cases, the Data Analyst is able to investigate documents' logic when he or she is fully on-boarded to the project and has been granted access to the documents, which takes time and is usually done on arrival. You may ask the DM to start this process in advance to save time.

note

Note that if the customer's documents include sensitive data, you need to observe security rules. Check the rules here.

Here are questions to be answered when studying documents:

  1. What is the quality of the original documents? Is it suitable for OCR? Can it be improved?
  2. What is the meaning of each class and field to be extracted from the documents?
  3. Where is each field given in the documents?
  4. Which fields are required and which are optional?
  5. Which fields are rarely given in the documents?
  6. Is there any special logic, complicated dependencies or exceptional cases?

Work with SMEs​

To organize work with SMEs in the most effective way, investigate the following questions:

  1. How many SMEs are available for labeling documents?
  2. Are they available full-time or part-time?
  3. What are their working hours?
  4. What are their names and emails?
  5. Who is the manager of the team and what are their contacts?

You may ask SMEs to fill in the following document, or fill it out on your own after discussion with SMEs.

Write an e-mail for the whole team with your analysis of the documents and SMEs availability.  Example of the email:

Hello team,

After the initial analysis of the data I would like to raise some issues that need to be investigated in advance to prevent their becoming pain spots:

1. Complex document type

There are purchase orders of supplier N that have table format and multiple pages. Here we can see that single-value fields (date, purchase order number and product) become multi-value fields, so we need a separate Manual Task for all the documents of that type. Most probably we will need to train a separate model for that or at least develop an extended model. For now there are 137 documents of that supplier, the info on volume of those documents and if there are similar cases has been requested from SMEs, waiting for response.

2. Rare field

The field "Oiling" seems to appear only in the documents of supplier X. There are only 17 documents of this supplier and not more can be provided, because it's a new supplier. The data for this field for all the other suppliers is being downloaded from Data Base. The value for this field is not constant, so there can be no guarantee that if the model trained on 17 documents will process this field in production with the required accuracy level (95%). The situation is to be talked over with the client.

3. SMEs availability

SMEs' team is not steady. Their manager told me that they are going to assign people each week depending on their business. That is an additional risk, because it will take extra time and effort to establish contact with them and train. We need to ask for a stable team of at least 3 SMEs with scheduled working hours at the project in order to label all the required Dataset of 1480 documents in time.

Best regards

  1. Potential risks not raised

    • Cases with documents of some specific structures, where one or more fields are given in the way not very simple for extraction, conflicting data)/ are there enough examples of them?
    • Cases that require training a separate model.
    • Cases of rare fields.
    • Small number of specific document structures (rare document types).

    Such complicated cases should be additionally investigated and raised to Delivery Manager as potential risks, if confirmed. Extra documents or resources may need to be requested from the customer, a special approach developed, or some complicated cases may be negotiated to be out of scope. When not reported during the early stages, there won't be enough time and resources to deal with them during later stages of the project.

  2. Logic is badly investigated

    When some dependencies or document and layout logic is not studied well, the wrong solution may be developed, for example, in labeling logic or labeling instructions or corner cases revealed during later stages of the project.

See also