Skip to main content

OCR how-tos

How to check license usage countdown​

Work.AI uses the so-called soft counter. For license counting, only the page count within a document is taken into account: 1 page = 1 license. The page size does not matter.

How to check maximum available units in license​

If you get the warning: "The maximum number of units available in the license has been reached", this means that you cannot continue the recognition because you have exceeded the page limit that your license allows you to recognize.

Use the activeLicense API call and check the following response attributes:

  • volumeRemaining: it can be equal to volume or 0. In the latter case, you need a new license. To get it, contact the WorkFusion Support service. For license activation instructions, refer to the topic.

  • volumeRefreshingPeriod: if the value is not infinite, then more pages are automatically included in the license once the refresh period is over.

For more information on API calls, refer to the OCR REST API topic.

How to check maximum number of concurrent recognition processes​

The number of recognition processes you can execute in parallel is limited to the number of cores allowed by the OCR license. The number of cores is defined by the allowedCoresCount attribute of the OCR license.

To get the information about the license, use the activeLicense API call. In the response, check the allowedCoresCount attribute. If its value is 0, the number of CPU cores is unlimited. See the OCR REST API topic.

How to check license consumption history​

Knowing the previous OCR consumption history is useful for planning the OCR licensing needs. To understand how many pages were consumed in the past weeks or months over a given period, try the following approach, depending on the Work.AI version you are using.

  1. Run the following request, depending on the ABBYY OCR engine version you are using:

    • For FREngine11: GET /api/v1/cloud/activeLicense

    • For FREngine12: GET /api/v2/cloud/activeLicense

    Mind that to run the command, you need SSH access to the OCR server. For more details, refer to Activate OCR license | Verify license.

  2. Check the volumeRemaining attribute.

  3. After some time, for example, one week, rerun the request and subtract the volumeRemaining attribute from the previous result.

You can also automate these actions, for instance, using a scheduled Business Process.

How to generate new licence_request.txt​

Get the license_request file from the OCR server and save it using the curl command below.

  • FREngine11:

    curl -X POST http://localhost:9002/api/v1/cloud/prepareLicense > INSTALL_DIR/ABBYY_FRE11/license_request.txt

  • FREngine12:

    curl -X POST http://localhost:9002/api/v2/cloud/prepareLicense > INSTALL_DIR/ABBYY_FRE11/license_request.txt

How to prevent OCR engine from removing handwritten parts​

warning

The instructions are valid only for Work.AI versions up to v10.2.9. Starting from 10.3, the allowedRegionTypes parameter is deprecated.

The allowedRegionTypes parameter specifies the allowed region types for the classification of identified blocks. If allowedRegionTypes = empty, all types are processed.

To keep information from handwritten parts, it is necessary to suppress the region classification as a picture of any type by specifying the parameter value as follows:

BT_Table,BT_Text,BT_Barcode,BT_Separator,BT_SeparatorGroup,BT_Checkmark,BT_CheckmarkGroup

note

No automatic handwriting recognition is applied. Information is rendered as an unreadable set of characters.

Example:

<http-param name="allowedRegionTypes">
        BT_Table,BT_Text,BT_Barcode,BT_Separator,BT_SeparatorGroup,BT_Checkmark,BT_CheckmarkGroup
</http-param>

How to improve poor image recognition quality​

Examples of poor recognition quality are a gray overlay or image cropped off during driver scan.

To improve the image recognition quality, use FRE 12 (OCR v2 API) and set the following parameter: enhanceLocalContrast=true. For more details, refer to OCR REST API and Improve OCR results.

How to resolve missed font issue​

If a recognized document contains dots instead of text, the OCR engine does not have required fonts.

To import Microsoft fonts into your system, run the following command from root on all Agent servers:

tar -x -C / -f INSTALL_DIR/ABBYY_FRE12/msttcore.tgz

How to convert searchable PDF into image-based one​

Convert a searchable PDF into an image-based one using the following transformation:

invoke(new ProcessBuilder(new String[] {
                        "convert",
            "-density","300",
                        inputFile.getAbsolutePath(),
                        result.getAbsolutePath()}
                    ));

The sample bot step below does the following:

  1. Downloads the original file to the file system.
  2. Converts it to an image-based PDF using ImageMagick.
  3. Uploads the new PDF file to S3.
  4. Removes temporary files.
  5. Exports the link to the new PDF to the next step.
See sample bot step code
<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">

    <required name="original_document_url"/>

    <var-def name="file_path">
        <script return="convert(original_document_url.toString())"><![CDATA[
            import java.io.BufferedReader;
            import java.io.File;
            import java.io.IOException;
            import java.io.InputStreamReader;
            import java.net.URL;
            import com.google.common.io.Files;
            import org.apache.commons.io.FileUtils;
            import org.apache.commons.io.FilenameUtils;

            static File outputFolder;

            static boolean getOS(){
                return ((String)System.getProperties().get("os.name")).contains("Windows");
            }

            boolean isWindows=getOS();

            static File convert(String documentLink) throws IOException, InterruptedException {
                File baseDir = Files.createTempDir();
                sys.defineVariable("tmpDirPath", baseDir.getPath());
                File inputFolder = new File(baseDir, "input");

                URL documentUrl = new URL(documentLink);
                String inputFileName = FilenameUtils.getName(documentUrl.getPath());

                if (inputFileName == null || inputFileName.isEmpty()) {
                    inputFileName = "input.pdf";
                }
                sys.defineVariable("inputFileName", inputFileName.substring(0,inputFileName.lastIndexOf(".")));

                File inputFile = new File(inputFolder, inputFileName);
                FileUtils.copyURLToFile(documentUrl, inputFile);

                outputFolder = new File(baseDir, "output");
                FileUtils.forceMkdir(outputFolder);

                File result = new File(outputFolder, "output.pdf");

                if (isWindows){
                    invoke(new ProcessBuilder(new String[] {
                        "magick",
                        "convert",
                        "-density","300",
                        inputFile.getAbsolutePath(),
                        result.getAbsolutePath()}
                    ));
                } else {
                    invoke(new ProcessBuilder(new String[] {
                        "convert",
                        "-density","300",
                        inputFile.getAbsolutePath(),
                        result.getAbsolutePath()}
                    ));
                }

                return result;
            }

            static void invoke(ProcessBuilder builder) throws IOException, InterruptedException {
                builder.redirectErrorStream(true);
                Process process = builder.start();

                BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
                String line;
                while ((line = in.readLine()) != null) {
                    log.info(line);
                }
                in.close();

                int code = process.waitFor();
                if (code != 0) {
                    log.error("Failed to invoke process: " + builder.command() + ". Return code: " + code);
                    throw new RuntimeException("Failed to invoke process: " + builder.command() + ". Return code: " + code);
                }
            }
        ]]></script>
    </var-def>

    <var-def name="content">
        <file path="${file_path}" type="binary"/>
    </var-def>

    <var-def name="converted_link">
        <s3  bucket="str">
            <s3-put-public path="converted/converted-${inputFileName}.pdf" content="${content}" content-type="application/pdf" content-disposition="inline"/>
        </s3>
    </var-def>

    <script><![CDATA[
        FileUtils.deleteDirectory(new File(tmpDirPath.toString()));
    ]]></script>

    <export include-original-data="true">
        <single-column name="converted_document_link" value="${converted_link}" />
    </export>
</config>
note

When processing a searchable PDF, extra spaces are added in words.

How to convert PDF to TIFF​

Use the converTo(value=tiff) parameter that detects the file type automatically and converts the file to TIFF (converts before processing). Accepted formats are PDF, PNG, JPG, JPEG. For more details, refer to the OCR REST API reference.

How to check maximum number of started concurrent recognition processes​

For the OCR Worker component, the ocr-worker.yml configuration file includes the worker.executor.abbyy.pool.size configuration property (ZooKeeper path: /config/ocr-worker). The property specifies the maximum number of concurrent recognition processes performed per OCR Worker instance. If it is not specified, the default value 2 is applied.

If you have a single OCR Worker instance, worker.executor.abbyy.pool.size must be equal to allowedCoresCount-1 so that one core remains free for license monitoring and health checks. If you have more OCR Worker servers, the sum of worker.executor.abbyy.pool.size for each OCR Worker must be equal to allowedCoresCount-1.

How to increase throughput per OCR server​

Consider increasing the number of available CPUs in your hardware. Once you upgrade, adjust the following property in the application configuration:

worker.executor.abbyy.pool.size=15 
# Where 15 is number of OCR license cores minus 1. Maximum is 32

How to correct page orientation​

For pages with incorrect orientation (rotated by 90 or 270 degrees), try correctOrientation=true. If that does not help, use exportFormat=xmlForCorrectedImage. For more details, refer to the OCR REST API reference.

The ocr-rest.yml configuration file for the OCR REST component includes three properties to configure the cleanup frequency:

# how long input documents or patterns are stored in S3;
# default value – 2000 minutes
db.cleanup.abbyy.input=2000
# how long output result is stored in S3, though metadata and inputs are not affected;
# default value – 2000 minutes
db.cleanup.abbyy.output=2000
# how long records are stored in DB (after that full cleanup in DB and S3);
# default value – 600 minutes
db.cleanup.abbyy.record=600

Once the smaller of these two has elapsed (in this case, 1,440 minutes = 1 day), the recognition result is deleted.

To keep the OCR result for a longer period, it is necessary to set an appropriate number. It means the OCR Service is not intended for storing large volumes of OCR inputs and outputs for a long time. If you need to increase the retention period to prevent from the negative impact on the OCR service performance, configure the service to use an S3 storage profile instead of the default gridfs-storage profile.