Combine desktop RPA approaches

Goal

The goal of the guide is to demonstrate the development approach of how to bring together:

• An ODF 2-based project started from the standard ODF 2 Archetype.
• Desktop RPA using WorkFusion Simplified RPA API.
• The Recorder OCR and Mouse Click actions. One of the Recorder's strengths is a fast and convenient UX for cutting screenshots and specifying areas.

Sample automation overview

Install applications

For the sample automation, you need two Windows desktop applications:

• Notepad: a standard simple Windows text editor.
• The FAR file manager: a legacy console-based file manager. It is selected because it emulates work with legacy "blue" mainframe AS400-like applications.

View scenario

1. Open Notepad and type "Now we will switch to FAR, navigate to C:/Users, and use OCR to capture area with a list of users and to fetch text."
2. Open FAR and navigate to C:/Users.
3. Use Recorder to cut an image with the area containing the list of users. For an anchor, use some FAR text shown on all cases. Generate the code using Recorder and modify it to work in the Java TransactionSupplier class.
4. Switch back to Notepad and type out the OCRed list of users.
5. In Notepad, type "Now we will switch back to FAR and click All Users using the Image as Target for the click."
6. Switch to FAR and click.
7. Use the list of the parsed users to set them as names of Documents for one Transaction.

Set up sample automation

1. Download example-rpa-notepad-far-automation.zip.
2. Unzip the archived sample.
3. Launch Recorder from Work.AI Developer.
4. Import a recorder script. In the Media Files panel, right-click and select Import > WorkFusion Recorder > Recorder Script and select the link to the attached example project to import.
5. Open the example project in IDEA IDE.
6. Install FAR Manager into the C:/FAR folder as hardcoded in the project's script. You can download the Far Manager v3.0 build 5700 x64 MSI installer for free. Make sure you use MSI and install it into C:/FAR.
7. Start OCR from Work.AI Developer, as the sample uses OCR.

Explore sample

1. You get the Maven project with two Maven modules and the Recorder project (recorder_project).

2. To open the Recorder project to learn, explore, and execute script actions, double-click the notepad-far-automation.rpae file.

3. In the rpa-notepad-far-automation-bcb module, review classes with the ODF 2 and RPA Java code:

   • org.example.supplier.TransactionSupplierRPAExample
   Expand the class

           package org.example.supplier;
   
           import java.util.ArrayList;
           import java.util.Collection;
           import java.util.List;
           import java.util.UUID;
           import java.util.stream.Collectors;
           import javax.inject.Inject;
   
           import groovy.lang.Binding;
           import org.example.rpa.FarRobot;
           import org.example.rpa.NotepadRobot;
   
           import com.workfusion.odf.api.connector.TransactionSupplier;
           import com.workfusion.odf.api.domain.Document;
           import com.workfusion.odf.api.domain.Transaction;
   
           public class TransactionSupplierRPAExample implements TransactionSupplier {
   
               private static final String FIRST_NOTEPAD_MESSAGE = "Now we will switch to FAR, navigate to C:/Users and use OCR \nto capture the area with a list of users and to fetch text\n\n";
               private static final String SECOND_NOTEPAD_MESSAGE = "\n\nNow we will switch back to FAR and click All Users \nusing Image as Target for the click";
           
               private final String ocrApiUrl;
   
               @Inject
               public TransactionSupplierRPAExample(Binding binding) {
                   //Get "ocrApiUrl" from binding
                   this.ocrApiUrl = binding.getVariable("ocrApiUrl").toString();
               }
   
               @Override
               public Collection<Transaction> get() {
                   //Create a transaction.
                   Collection<Transaction> transactions = new ArrayList<>();
                   Transaction transaction = new Transaction();
                   transaction.setId(uuid());
                   transaction.setDocs(createDocuments());
                   transactions.add(transaction);
   
                   return transactions;
               }
   
               private List<Document> createDocuments() {
                   //Run Notepad and print the first message.
                   NotepadRobot notepad = new NotepadRobot();
                   notepad.openNotepad();
                   notepad.printMessage(FIRST_NOTEPAD_MESSAGE);
   
                   //Run FAR, navigate to C:/Users, and use OCR to capture the area with the list of users.
                   FarRobot far = new FarRobot("C:\\FAR");
                   far.openFar();
                   final List<String> users = far.recognizeUsers(ocrApiUrl);
   
                   //Print the list of users and the second message to Notepad.
                   notepad.printMessage(String.format("Users: %s", String.join(", ", users)));
                   notepad.printMessage(SECOND_NOTEPAD_MESSAGE);
   
                   //Double-click on All Users using Image as Target for click.
                   far.doubleClickAllUsersFolderByImage();
   
                   //Close FAR and Notepad.
                   far.close();
                   notepad.close();
   
                   //Create a list of documents with usernames.
                   return users.stream().map(this::createDocument).collect(Collectors.toList());
               }
   
               private Document createDocument(String userName) {
                   final Document document = new Document();
                   document.setId(uuid());
                   document.putAttribute("user_name", userName);
                   return document;
               }
   
               private String uuid() {
                   return UUID.randomUUID().toString();
               }
   
           }
   • org.example.rpa.FarRobot
   Expand the class

           package org.example.rpa;
   
           import java.util.Arrays;
           import java.util.List;
   
           import org.openqa.selenium.Rectangle;
   
           import com.workfusion.rpa.helpers.Ocr;
           import com.workfusion.rpa.helpers.RPA;
           import com.workfusion.studio.rpa.recorder.api.StringTransformations;
           import com.workfusion.studio.rpa.recorder.api.WindowDescriptor;
   
           import static com.workfusion.rpa.helpers.RPA.$;
           import static com.workfusion.rpa.helpers.RPA.pressEnter;
           import static com.workfusion.rpa.helpers.RPA.sendKeys;
           import static com.workfusion.rpa.helpers.RPA.switchToExistingWindow;
           import static com.workfusion.rpa.helpers.RPA.title;
           import static com.workfusion.rpa.helpers.UiSelectors.byImage;
   
           public class FarRobot {
               
               // Modify those URLs to point to the APNG files you put into your file storage
               private final String FAR_IMAGE_OCR_LINK = "http://localhost:15110/doc-upload/1614612014683-anchor-1614612014912.apng";
               private final String FAR_IMAGE_CLICK_LINK = "http://localhost:15110/doc-upload/1614612014683-anchor-1614614100259.apng";
   
               private final String farPath;
   
               private String title;
   
               public FarRobot(String farPath) {
                   this.farPath = farPath;
               }
   
               public void openFar() {
                   sendKeys(StringTransformations.getHotKeyText(114, 4));
                   switchToExistingWindow(new WindowDescriptor("#32770", "Run", false, false).toString(), 10000L);
                   sendKeys("C:\\FAR\\Far.exe");
                   pressEnter();
                   
                   switchToExistingWindow(new WindowDescriptor("ConsoleWindowClass", String.format("{%s} - Far 3.0.5700.0 x64", farPath.replaceAll(
                           "\\\\Far[.]exe", "")), false, false).toString(), 10000L);
                   this.title = title();
               }
   
               public void changeDirectory(String path) {
                   switchToFar();
                   sendKeys(String.format("cd %s", path));
                   pressEnter();
   
                   this.title = title();
               }
   
               public List<String> recognizeUsers(String ocrApiUrl) {
                   if (!title.contains("C:\\Users")) {
                       changeDirectory("C:\\Users");
                   }
   
                   final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();
                   final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -120, rectangle.y + 60, 194, 172);
   
                   return Arrays.asList(ocrResult.split(System.lineSeparator()));
               }
   
               public void doubleClickAllUsersFolderByImage() {
                   switchToFar();
                   if (!title.contains("C:\\Users")) {
                       changeDirectory("C:\\Users");
                   }
   
                   $(byImage(FAR_IMAGE_CLICK_LINK, -189, 91)).doubleClick();
                   RPA.sleep(3000);
                   this.title = title();
               }
   
               public void close() {
                   switchToFar();
                   RPA.close();
               }
   
               private void switchToFar() {
                   if (!title().equals(title)) {
                       switchToExistingWindow(new WindowDescriptor("ConsoleWindowClass", title, false, false).toString(), 10000L);
                   }
               }
   
           }
   • org.example.rpa.NotepadRobot
   Expand the class

           package org.example.rpa;
   
           import com.workfusion.rpa.helpers.RPA;
           import com.workfusion.studio.rpa.recorder.api.WindowDescriptor;
   
           import static com.workfusion.rpa.helpers.RPA.$;
           import static com.workfusion.rpa.helpers.RPA.open;
           import static com.workfusion.rpa.helpers.RPA.pressEnter;
           import static com.workfusion.rpa.helpers.RPA.sendKeys;
           import static com.workfusion.rpa.helpers.RPA.switchToExistingWindow;
           import static com.workfusion.rpa.helpers.RPA.title;
   
           public class NotepadRobot {
   
               private String title;
   
               public void openNotepad() {
                   this.title = "Untitled - Notepad";
   
                   open("notepad");
                   switchToExistingWindow(new WindowDescriptor("Notepad", title, false, false).toString(), 10000L);
               }
   
               public void printMessage(String message) {
                   switchToNotepad();
                   $("[CLASS:Edit]").click();
                   sendKeys(message);
                   pressEnter();
   
                   this.title = title();
               }
   
               public void close() {
                   switchToNotepad();
                   RPA.close();
                   switchToExistingWindow(new WindowDescriptor("#32770", "Notepad", false, false).toString(), 10000L);
                   $("[CLASS:Button; NAME:CommandButton_7]").click();
               }
   
               private void switchToNotepad() {
                   if (!title().equals(title)) {
                       switchToExistingWindow(new WindowDescriptor("Notepad", title, false, false).toString(), 10000L);
                   }
               }
   
           }

4. In IDEA IDE, find and run Transaction Supplier RPA Example.xml to execute the example.

   

Reuse generated code in ODF 2

Prepare APNG files

You can get ANPNG files from the Recorder project or from the image folder in the generated project. These files contain a screenshot and the position anchor region.

To access APNG files from Control Tower, upload them to S3 Manager and use the generated link as the file path.

S3 Manager is a part of Work.AI Developer installed on your workstation. To use it, perform the following steps:

1. Open Launcher and start the Control Tower or Workspace group.
2. Navigate to S3 Manager. To log in, use the credentials you have during the Work.AI Developer installation.
3. Upload files and generate URLs for both.
4. Type URLs into the org.example.rpa.FarRobot class.

In the project, links to files are presented in the FarRobot class as the FAR_IMAGE_OCR_LINK and FAR_IMAGE_CLICK_LINK variables.

The FAR manager can look differently for different OS versions. To make the sample work, switch to Recorder, find the OCR action (line 12), and capture a new image of FAR in your system.

A similar issue can occur for the Mouse Click action (line 21). Add new APNG images to S3 Manager, generate their URLs, and use them in rpatestodf2.task.FarRobot.

Perform OCR step

The generated OCR code looks like this:

org.openqa.selenium.Rectangle i0 = $(byImage("C:\\Users\\username\\workfusion-workspace\\rpa-notepad-far-automation\\images\\1614249595032-anchor-1614249595183.apng")).getRect()
ocr_result = RString.of(Ocr.proccessImage(sysOcrApiUrl.toString(), "English", i0.x + -92, i0.y + 32, 229, 272))

As you can see, to get the OCR result, you have to get the Rectangle object from the APNG file and call the OCR API with the OCR language and coordinates.

After you upload the required files to S3 and add a method to get the path to the resource file, you can create a Rectangle object using the following code:

final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();

The second value you should get is ocrApiUrl. It is stored in the binding values. To access the binding, inject it into the supplier. The following code shows how to get ocrApiUrl:

@Inject
public TransactionSupplierRPAExample(Binding binding) {
    //Get "ocrApiUrl" from binding
    this.ocrApiUrl = binding.getVariable("ocrApiUrl").toString();
}

Now, you have all required variables to call OCR from ODF 2:

final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -92, rectangle.y + 32, 229, 272);

If you compare the generated code and the ODF 2 code, you can see that they are similar.

Let's go back to the sample ODF 2 project and look at the OCR step implementation. The recognizeUsers method in the FarRobot class recognizes the username in the C:/Users folder from the FAR screen and returns a list of usernames.

public List<String> recognizeUsers(String ocrApiUrl) {
    if (!title.contains("C:\\Users")) {
        changeDirectory("C:\\Users");
    }

    final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();
    final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -92, rectangle.y + 32, 229, 272);

    return Arrays.asList(ocrResult.split(System.lineSeparator()));
}

Perform click-by-image step

The generated code for the click-by-image step looks like this:

$(byImage("C:\\Users\\username\\workfusion-workspace\\rpa-notepad-far-automation\\images\\1614249595032-anchor-1614251201543.apng", Integer.valueOf(-193), Integer.valueOf(41))).doubleClick()

As you can see, in the step, you use the byImage method, but in this case, the method receives two additional parameters: the coordinates for the click. To implement this step in your project, use the following code:

$(byImage(FAR_IMAGE_CLICK_LINK, -193, 41)).doubleClick();

Find the implementation of this action in the doubleClickAllUsersFolderByImage method in the FarRobot class:

public void doubleClickAllUsersFolderByImage() {
    switchToFar();
    if (!title.contains("C:\\Users")) {
        changeDirectory("C:\\Users");
    }

    $(FAR_IMAGE_CLICK_LINK, -193, 41)).doubleClick();
    RPA.sleep(1000);
    this.title = title();
}