OCR how-tos
How to check license usage countdown
Intelligent Automation Cloud Enterprise uses the so-called soft counter. For license counting, only the page count within a document is taken into account: 1 page = 1 license. The page size does not matter.
How to check maximum available units in license
If you get the warning: "The maximum number of units available in the license has been reached", this means that you cannot continue the recognition because you have exceeded the page limit that your license allows you to recognize.
Use the activeLicense API call and check the following response attributes:
volumeRemaining: it can be equal tovolumeor0. In the latter case, you need a new license. To get it, contact the WorkFusion Support service. For license activation instructions, refer to the topic.volumeRefreshingPeriod: if the value is notinfinite, then more pages are automatically included in the license once the refresh period is over.
For more information on API calls, refer to the OCR REST API topic.
How to check maximum number of concurrent recognition processes
The number of recognition processes you can execute in parallel is limited to the number of cores allowed by the OCR license. The number of cores is defined by the allowedCoresCount attribute of the OCR license.
To get the information about the license, use the activeLicense API call. In the response, check the allowedCoresCount attribute. If its value is 0, the number of CPU cores is unlimited. See the OCR REST API topic.
How to check license consumption history
Knowing the previous OCR consumption history is useful for planning the OCR licensing needs. To understand how many pages were consumed in the past weeks or months over a given period, try the following approach:
Run the
GET /api/v1/cloud/activeLicenserequest and check thevolumeRemainingattribute.Mind that to run the command, you need SSH access to the OCR server. For more details, refer to Activate OCR license | Verify license.
After some time, for example, one week, rerun the request and subtract the
volumeRemainingattribute from the previous result.
You can also automate these actions, for instance, using a scheduled Business Process.
How to generate new licence_request.txt
Get the license_request file from the OCR server and save it using the curl command below:
curl -X POST http://localhost:9002/api/v1/cloud/prepareLicense > INSTALL_DIR/ABBYY_FRE11/license_request.txt
How to avoid removing handwritten parts removed by OCR engine
The allowedRegionTypes parameter specifies the allowed region types for the classification of identified blocks. If allowedRegionTypes = empty, all types are processed.
To keep information from handwritten parts, it is necessary to suppress the region classification as a picture of any type by specifying the parameter value as follows:
BT_Table,BT_Text,BT_Barcode,BT_Separator,BT_SeparatorGroup,BT_Checkmark,BT_CheckmarkGroup
note
No automatic handwriting recognition is applied. Information is rendered as an unreadable set of characters.
Example:
<http-param name="allowedRegionTypes">
BT_Table,BT_Text,BT_Barcode,BT_Separator,BT_SeparatorGroup,BT_Checkmark,BT_CheckmarkGroup
</http-param>
How to improve poor image recognition quality
Examples of poor recognition quality are a gray overlay or image cropped off during driver scan.
To improve the image recognition quality, use FRE 12 (OCR v2 API) and set the following parameter: enhanceLocalContrast=true. For more details, refer to OCR REST API and Improve OCR results.
How to resolve missed fonts issue
If a recognized document contains dots instead of text, the OCR engine does not have required fonts.
To import Microsoft fonts into your system, run the following command from root on all Agent servers:
tar -x -C / -f INSTALL_DIR/ABBYY_FRE12/msttcore.tgz
How to convert searchable PDF into image-based one
Convert a searchable PDF into an image-based one using the following transformation:
invoke(new ProcessBuilder(new String[] {
"convert",
"-density","300",
inputFile.getAbsolutePath(),
result.getAbsolutePath()}
));
The full bot step below does the following:
- Downloads the original file to the file system.
- Converts it to an image-based PDF using ImageMagick.
- Uploads the new PDF file to S3.
- Removes temporary files.
- Exports the link to the new PDF to the next step.
<?xml version="1.0" encoding="UTF-8"?>
<config charset="UTF-8">
<required name="original_document_url"/>
<var-def name="file_path">
<script return="convert(original_document_url.toString())"><![CDATA[
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import com.google.common.io.Files;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;
static File outputFolder;
static boolean getOS(){
return ((String)System.getProperties().get("os.name")).contains("Windows");
}
boolean isWindows=getOS();
static File convert(String documentLink) throws IOException, InterruptedException {
File baseDir = Files.createTempDir();
sys.defineVariable("tmpDirPath", baseDir.getPath());
File inputFolder = new File(baseDir, "input");
URL documentUrl = new URL(documentLink);
String inputFileName = FilenameUtils.getName(documentUrl.getPath());
if (inputFileName == null || inputFileName.isEmpty()) {
inputFileName = "input.pdf";
}
sys.defineVariable("inputFileName", inputFileName.substring(0,inputFileName.lastIndexOf(".")));
File inputFile = new File(inputFolder, inputFileName);
FileUtils.copyURLToFile(documentUrl, inputFile);
outputFolder = new File(baseDir, "output");
FileUtils.forceMkdir(outputFolder);
File result = new File(outputFolder, "output.pdf");
if (isWindows){
invoke(new ProcessBuilder(new String[] {
"magick",
"convert",
"-density","300",
inputFile.getAbsolutePath(),
result.getAbsolutePath()}
));
} else {
invoke(new ProcessBuilder(new String[] {
"convert",
"-density","300",
inputFile.getAbsolutePath(),
result.getAbsolutePath()}
));
}
return result;
}
static void invoke(ProcessBuilder builder) throws IOException, InterruptedException {
builder.redirectErrorStream(true);
Process process = builder.start();
BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
log.info(line);
}
in.close();
int code = process.waitFor();
if (code != 0) {
log.error("Failed to invoke process: " + builder.command() + ". Return code: " + code);
throw new RuntimeException("Failed to invoke process: " + builder.command() + ". Return code: " + code);
}
}
]]></script>
</var-def>
<var-def name="content">
<file path="${file_path}" type="binary"/>
</var-def>
<var-def name="converted_link">
<s3 bucket="str">
<s3-put-public path="converted/converted-${inputFileName}.pdf" content="${content}" content-type="application/pdf" content-disposition="inline"/>
</s3>
</var-def>
<script></script>
<export include-original-data="true">
<single-column name="converted_document_link" value="${converted_link}" />
</export>
</config>
note
When processing a searchable PDF, extra spaces are added in words.
How to convert PDF to TIFF or ImageMagick
Use the converTo(value=tiff) parameter that detects the file type automatically and converts the file to TIFF (converts before processing). Accepted formats are PDF, PNG, JPG, JPEG. For more details, refer to the OCR REST API reference.
How to check maximum number of started concurrent recognition processes
For the OCR Worker component, the ocr-worker.yml configuration file includes the worker.executor.abbyy.pool.size configuration property (ZooKeeper path: /config/ocr-worker). It specifies the maximum number of concurrent recognition processes performed per OCR Worker instance. If it is not specified, the default value 2 is applied.
If you have a single OCR Worker instance, worker.executor.abbyy.pool.size must be equal to allowedCoresCount-1 so that one core remains free for license monitoring and health checks. If you have more OCR Worker servers, the sum of worker.executor.abbyy.pool.size for each OCR Worker must be equal to allowedCoresCount-1.
How to increase throughput per OCR server
Consider increasing the number of available CPUs in your hardware. Once you upgrade, adjust the following property in the application configuration:
worker.executor.abbyy.pool.size=15
# Where 15 is number of OCR license cores minus 1. Maximum is 32
How to correct page orientation
For pages with incorrect orientation (rotated by 90 or 270 degrees), if correctOrientation=true does not help, try using exportFormat=xmlForCorrectedImage. For more details, refer to the OCR REST API reference.
How to increase OCR result link expiration term
For Intelligent Automation Cloud Enterprise
The ocr-rest.yml configuration file for the OCR REST component includes three properties to configure the cleanup frequency:
# how long input documents or patterns are stored in S3;
# default value – 2000 minutes
db.cleanup.abbyy.input=2000
# how long output result is stored in S3, though metadata and inputs are not affected;
# default value – 2000 minutes
db.cleanup.abbyy.output=2000
# how long records are stored in DB (after that full cleanup in DB and S3);
# default value – 600 minutes
db.cleanup.abbyy.record=600
Once the smaller of these two has elapsed (in this case, 1,440 minutes = 1 day), the recognition result is deleted.
To keep the OCR result for a longer period, it is necessary to set an appropriate number. It means the OCR Service is not intended for storing large volumes of OCR inputs and outputs for a long time. If you need to increase the retention period to prevent from the negative impact on the OCR service performance, configure the service to use an S3 storage profile instead of the default gridfs-storage profile.
For Intelligent Automation Cloud Business
OCR results are deleted not based on the storage time but on the number of stored OCR "tasks".
ocr.tasks.abbyy.storage.capacity=50
For Intelligent Automation Cloud Express (Business), the number is configured in application.properties of the OCR service. OCR inputs and outputs are stored in memory, so it's not recommended to set the value too high. For example, if the sum of the average input image size and exported document size is equal to 1 MB and you set the capacity to 1,000, it will consume approximately 1 GB of your RAM.
note
As this data is kept in memory, it is lost as soon as you shut down or restart the OCR service.
View also: