10.1.7

Edit

Mixed approaches in desktop RPA

Goal

The goal of the guide is to demonstrate the development approach of how to bring together:

  • An ODF-based project started from the standard ODF Archetype.
  • Desktop RPA using WorkFusion Simplified RPA API.
  • The Recorder OCR and Mouse Click actions. One of the Recorder's strengths is a fast and convenient UX for cutting screenshots and specifying areas.

Sample automation overview

Install applications

For the sample automation, you need two Windows desktop applications:

  • Notepad——a standard simple Windows text editor.
  • The FAR file manager—a legacy console-based file manager. It is selected because it emulates work with legacy "blue" mainframe AS400-like applications.

View scenario

  1. Open Notepad and type "Now we will switch to FAR, navigate to C:/Users, and use OCR to capture area with a list of users and to fetch text."
  2. Open FAR and navigate to C:/Users.
  3. Use Recorder to cut an image with the area containing the list of users. For an anchor, use some FAR text shown on all cases. Generate the code using Recorder and modify it to work in the Java TransactionSupplier class.
  4. Switch back to Notepad and type out the OCRed list of users.
  5. In Notepad, type "Now we will switch back to FAR and click All Users using the Image as Target for the click."
  6. Switch to FAR and click.
  7. Use the list of the parsed users to set them as names of Documents for one ODF Transaction.

Set up sample automation

  1. Download rpa-notepad-far-automation.zip.
  2. Unzip the archived sample.
  3. Launch WorkFusion Studio IDE from IA Cloud Developer.
  4. In the WorkFusion Project Navigator panel, right-click and select Import > Maven > Existing Maven Projects > select the root folder of the unzipped archive and choose both Maven modules to import.
  5. Install FAR manager into C:\FAR folder as hardcoded in the project's script. You can download the Far Manager v3.0 build 5700 x64 MSI installer for free. Make sure you use MSI and install it into C:\FAR.
  6. Start OCR from IA Cloud Developer as the sample uses OCR.

Explore sample

  1. You get the Maven project with two Maven modules and the Recorder project (recorder_project).

  2. To open the Recorder project to learn, explore, and execute script actions, double-click the notepad-far-automation.rpae file.

  3. In the rpa-notepad-far-automation-bcb module, review classes with the ODF and RPA Java code:

    1. org.example.supplier.TransactionSupplierRPAExample

      Expand the class

          package org.example.supplier;

    import java.util.ArrayList;
    import java.util.Collection;
    import java.util.List;
    import java.util.UUID;
    import java.util.stream.Collectors;
    import javax.inject.Inject;

    import groovy.lang.Binding;
    import org.example.rpa.FarRobot;
    import org.example.rpa.NotepadRobot;

    import com.workfusion.odf.api.connector.TransactionSupplier;
    import com.workfusion.odf.api.domain.Document;
    import com.workfusion.odf.api.domain.Transaction;

    public class TransactionSupplierRPAExample implements TransactionSupplier {

        private static final String FIRST_NOTEPAD_MESSAGE = "Now we will switch to FAR, navigate to C:/Users and use OCR \nto capture the area with a list of users and to fetch text\n\n";
        private static final String SECOND_NOTEPAD_MESSAGE = "\n\nNow we will switch back to FAR and click All Users \nusing Image as Target for the click";
    
        private final String ocrApiUrl;

        @Inject
        public TransactionSupplierRPAExample(Binding binding) {
            //Get "ocrApiUrl" from binding
            this.ocrApiUrl = binding.getVariable("ocrApiUrl").toString();
        }

        @Override
        public Collection<Transaction> get() {
            //Create a transaction.
            Collection<Transaction> transactions = new ArrayList<>();
            Transaction transaction = new Transaction();
            transaction.setId(uuid());
            transaction.setDocs(createDocuments());
            transactions.add(transaction);

            return transactions;
        }

        private List<Document> createDocuments() {
            //Run Notepad and print the first message.
            NotepadRobot notepad = new NotepadRobot();
            notepad.openNotepad();
            notepad.printMessage(FIRST_NOTEPAD_MESSAGE);

            //Run FAR, navigate to C:/Users, and use OCR to capture the area with the list of users.
            FarRobot far = new FarRobot("C:\\FAR");
            far.openFar();
            final List<String> users = far.recognizeUsers(ocrApiUrl);

            //Print the list of users and the second message to Notepad.
            notepad.printMessage(String.format("Users: %s", String.join(", ", users)));
            notepad.printMessage(SECOND_NOTEPAD_MESSAGE);

            //Double-click on All Users using Image as Target for click.
            far.doubleClickAllUsersFolderByImage();

            //Close FAR and Notepad.
            far.close();
            notepad.close();

            //Create a list of documents with usernames.
            return users.stream().map(this::createDocument).collect(Collectors.toList());
        }

        private Document createDocument(String userName) {
            final Document document = new Document();
            document.setId(uuid());
            document.putAttribute("user_name", userName);
            return document;
        }

        private String uuid() {
            return UUID.randomUUID().toString();
        }

    }

    2. org.example.rpa.FarRobot

      Expand the class

          package org.example.rpa;

    import java.util.Arrays;
    import java.util.List;

    import org.openqa.selenium.Rectangle;

    import com.workfusion.rpa.helpers.Ocr;
    import com.workfusion.rpa.helpers.RPA;
    import com.workfusion.studio.rpa.recorder.api.StringTransformations;
    import com.workfusion.studio.rpa.recorder.api.WindowDescriptor;

    import static com.workfusion.rpa.helpers.RPA.$;
    import static com.workfusion.rpa.helpers.RPA.pressEnter;
    import static com.workfusion.rpa.helpers.RPA.sendKeys;
    import static com.workfusion.rpa.helpers.RPA.switchToExistingWindow;
    import static com.workfusion.rpa.helpers.RPA.title;
    import static com.workfusion.rpa.helpers.UiSelectors.byImage;

    public class FarRobot {
        
        // Modify those URLs to point to the APNG files you put into your file storage
        private final String FAR_IMAGE_OCR_LINK = "http://localhost:15110/doc-upload/1614612014683-anchor-1614612014912.apng";
        private final String FAR_IMAGE_CLICK_LINK = "http://localhost:15110/doc-upload/1614612014683-anchor-1614614100259.apng";

        private final String farPath;

        private String title;

        public FarRobot(String farPath) {
            this.farPath = farPath;
        }

        public void openFar() {
            sendKeys(StringTransformations.getHotKeyText(114, 4));
            switchToExistingWindow(new WindowDescriptor("#32770", "Run", false, false).toString(), 10000L);
            sendKeys("C:\\FAR\\Far.exe");
            pressEnter();
            
            switchToExistingWindow(new WindowDescriptor("ConsoleWindowClass", String.format("{%s} - Far 3.0.5700.0 x64", farPath.replaceAll(
                    "\\\\Far[.]exe", "")), false, false).toString(), 10000L);
            this.title = title();
        }

        public void changeDirectory(String path) {
            switchToFar();
            sendKeys(String.format("cd %s", path));
            pressEnter();

            this.title = title();
        }

        public List<String> recognizeUsers(String ocrApiUrl) {
            if (!title.contains("C:\\Users")) {
                changeDirectory("C:\\Users");
            }

            final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();
            final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -120, rectangle.y + 60, 194, 172);

            return Arrays.asList(ocrResult.split(System.lineSeparator()));
        }

        public void doubleClickAllUsersFolderByImage() {
            switchToFar();
            if (!title.contains("C:\\Users")) {
                changeDirectory("C:\\Users");
            }

            $(byImage(FAR_IMAGE_CLICK_LINK, -189, 91)).doubleClick();
            RPA.sleep(3000);
            this.title = title();
        }

        public void close() {
            switchToFar();
            RPA.close();
        }

        private void switchToFar() {
            if (!title().equals(title)) {
                switchToExistingWindow(new WindowDescriptor("ConsoleWindowClass", title, false, false).toString(), 10000L);
            }
        }

    }

    3. org.example.rpa.NotepadRobot

      Expand the class

          package org.example.rpa;

    import com.workfusion.rpa.helpers.RPA;
    import com.workfusion.studio.rpa.recorder.api.WindowDescriptor;

    import static com.workfusion.rpa.helpers.RPA.$;
    import static com.workfusion.rpa.helpers.RPA.open;
    import static com.workfusion.rpa.helpers.RPA.pressEnter;
    import static com.workfusion.rpa.helpers.RPA.sendKeys;
    import static com.workfusion.rpa.helpers.RPA.switchToExistingWindow;
    import static com.workfusion.rpa.helpers.RPA.title;

    public class NotepadRobot {

        private String title;

        public void openNotepad() {
            this.title = "Untitled - Notepad";

            open("notepad");
            switchToExistingWindow(new WindowDescriptor("Notepad", title, false, false).toString(), 10000L);
        }

        public void printMessage(String message) {
            switchToNotepad();
            $("[CLASS:Edit]").click();
            sendKeys(message);
            pressEnter();

            this.title = title();
        }

        public void close() {
            switchToNotepad();
            RPA.close();
            switchToExistingWindow(new WindowDescriptor("#32770", "Notepad", false, false).toString(), 10000L);
            $("[CLASS:Button; NAME:CommandButton_7]").click();
        }

        private void switchToNotepad() {
            if (!title().equals(title)) {
                switchToExistingWindow(new WindowDescriptor("Notepad", title, false, false).toString(), 10000L);
            }
        }

    }

  4. To execute the example, right-click Transaction Supplier RPA Example.xml and select Run As > Bot Task.

Prepare Recorder project

As you know, Recorder can export code to the Groovy code. In the example, see how to use the generated code in ODF. Prepare to use the code of generated OCR and click-by-image steps in ODF:

  1. Create required steps in Recorder. In the recorder_project folder, find a Recorder project.

  2. Check that the process works fine from Recorder. For correct work of the sample script, in the far_path variable, set a path to Far.exe. If you have a problem with OCR or clicking by image step, recreate screenshots:

  3. Export the Recorder project code to a Bot Task.

Reuse generated code in ODF

Prepare APNG files

You can get ANPNG files from the Recorder project or the image folder in the generated project. These files contain a screenshot and the position anchor region.

To access APNG files from Control Tower, upload them to S3 (Minio) and use the link as the file path.

The Minio file storage is a part of IA Cloud Developer installed on your workstation. To use it, perform the following steps:

  1. Start Server Components in IA Cloud Developer.
  2. Navigate to http://localhost:15110/minio/.To log in, use the credentials you have set during the IA Cloud Developer installation.
  3. Upload files and generate URLs for both.
  4. Type URLs into the org.example.rpa.FarRobot class.

In the project, links to files are presented in the FarRobot class as the FAR_IMAGE_OCR_LINK and FAR_IMAGE_CLICK_LINK variables.

The FAR manager can look differently for different OS versions. To make the sample work, switch to the Recorder perspective, find the OCR action (line 12), and capture a new image of FAR in your system.

A similar issue can occur for the Mouse Click action (line 21). Add new APNG images to Minio, generate their URLs, and use them in org.example.rpa.FarRobot.

Perform OCR step

The generated OCR code looks like this:

org.openqa.selenium.Rectangle i0 = $(byImage("C:\\Users\\username\\workfusion-workspace\\rpa-notepad-far-automation\\images\\1614249595032-anchor-1614249595183.apng")).getRect()
ocr_result = RString.of(Ocr.proccessImage(sysOcrApiUrl.toString(), "English", i0.x + -92, i0.y + 32, 229, 272))

As you can see, to get the OCR result, you have to get the Rectangle object from the APNG file and call the OCR API with the OCR language and coordinates.

After you upload the required files to S3 and add a method to get the path to the resource file, you can create a Rectangle object using the following code:

final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();

The second value you should get is ocrApiUrl. It is stored in the binding values. To access the binding, inject it into the supplier. The following code shows how to get ocrApiUrl:

@Inject
public TransactionSupplierRPAExample(Binding binding) {
    //Get "ocrApiUrl" from binding
    this.ocrApiUrl = binding.getVariable("ocrApiUrl").toString();
}

Now, you have all required variables to call OCR from ODF:

final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -92, rectangle.y + 32, 229, 272);

If you compare the generated code and the ODF code, you can see that they are similar.

Let's go back to the sample ODF project and look at the OCR step implementation. The recognizeUsers method in the FarRobot class recognizes the username in the C:\Users folder from the FAR screen and returns a list of usernames.

public List<String> recognizeUsers(String ocrApiUrl) {
    if (!title.contains("C:\\Users")) {
        changeDirectory("C:\\Users");
    }

    final Rectangle rectangle = $(byImage(FAR_IMAGE_OCR_LINK)).getRect();
    final String ocrResult = Ocr.proccessImage(ocrApiUrl, "English", rectangle.x + -92, rectangle.y + 32, 229, 272);

    return Arrays.asList(ocrResult.split(System.lineSeparator()));
}

Perform click-by-image step

The generated code for the click-by-image step looks like this:

$(byImage("C:\\Users\\username\\workfusion-workspace\\rpa-notepad-far-automation\\images\\1614249595032-anchor-1614251201543.apng", Integer.valueOf(-193), Integer.valueOf(41))).doubleClick()

As you can see, in the step, you use the byImage method, but in this case, the method receives two additional parameters: the coordinates for the click. To implement this step in an ODF project, use the following code:

$(byImage(FAR_IMAGE_CLICK_LINK, -193, 41)).doubleClick();

Find the implementation of this action in the doubleClickAllUsersFolderByImage method in the FarRobot class:

public void doubleClickAllUsersFolderByImage() {
    switchToFar();
    if (!title.contains("C:\\Users")) {
        changeDirectory("C:\\Users");
    }

    $(FAR_IMAGE_CLICK_LINK, -193, 41)).doubleClick();
    RPA.sleep(1000);
    this.title = title();
}
← RPA with PageObjectLearn Java examples →