Inspect tools for desktop automation

Need of third party tools in RPA

Web applications do not always confine themselves to working entirely on the web. Sometimes they need to interact with the desktop to do things like downloads & uploads. Automating these sorts of workflow is tricky in using Web Driver. Web driver is confined to automating browsers, so desktop windows are out of scope.

This article describes two main tools that leverage desktop automation by providing information from a specified window (title, class, instance) and all its controls (edit boxes, check boxes, list boxes, combos, buttons, status bars):

AutoIt Window Info: for standard Windows applications
JVM Inspector: for Java windows’ controls (for example, applets)

AutoIt window info tool

AutoIt Window Info tool is useful when you need to find necessary information about application windows' controls, status bars, edit boxes and other elements. This information is used in RPA configurations to interact with windows’ elements.

Also, refer to the official documentation.

Go to C:\RPA\autoit\ folder.
Run Au3Info.exe (32-bit Windows) or Au3Info_x64.exe (64-bit Windows). The following window should appear on the screen:
Find the crosshair icon in the Finder Tool box and drag it to the object in which you are interested. In this example we use Windows Calculator and we are looking for information about the calculate "=" button.
You may see that AutoIt Window Info tool has populated all the information, which is required for AutoIt configuration. There are two main sections: Basic Window Info and Basic Control Info:

The first one (Basic Window Info) contains information about Calculator’s window. This information is used to switch to Calculator’s window:

window("[CLASS:CalcFrame]")

The second section (Basic Control Info) contains information about the button we have targeted with Finder Tool. Using this information you are able to click the "=" button:

$("[CLASS:Button; INSTANCE:28]").click()

For more detailed information about AutoIt and AutoIt Window Info tool refer to this article.

You can copy values to clipboard by double-clicking them.

JVM Inspector

JVM Inspector is a special Java application, which helps you find descriptors for Java windows’ controls (e.g. applets).

To start the JVM Inspector, open the ****C:\RPA\jvm-inspector**** folder and run the start-jvm-inspector.bat file.

Working with Component Tree

On the left side of JVM inspector window, you can find two sections: the top one shows active JVM processes and the bottom one shows window’s components.On the right side, you can find the Search field, Unique Descriptor text field and information about the selected component in the form of a table.

When some JVM processes are running, you could find them in the left top tree: Virtual Machines.

If you launched your applet after launching the JVM Inspector, click the Refresh virtual machines button.

If something is changed with JVM process or current JVM window (e.g. applet) you may use Refresh virtual machines button and Refresh UI tree button to update“JVM Inspector to the current state.

To connect to some JVM window, expand target process (7928: sun.plugin2... ** on the screen below) and double click the child window (JFrame [0x000C07EE]** on the screen below).
Now you can see that the bottom left tree (The Component Tree) has been updated and shows all the components of the selected window.
Select a node in the Component Tree, and you will find information about this particular selected component on the right side of JVM Inspector. Pay attention to unique descriptor field. This descriptor can be copied to your machine configuration.
The target UI control will be highlighted.

Using Search for Testing Descriptors

On the right side of the JVM Inspector, you can also find a Search control which is used to test your descriptor and find out if it works before you put it to a machine configuration. To test a descriptor:

Copy your descriptor, for example [CLASS: JCheckBox; TEXT: Northern], to the Search for component… text field.
You can also use a descriptor from the Unique descriptor text field for testing purposes.
Click the Search button.
If the provided descriptor exists, JVM Inspector will find the component according to the provided descriptor and select it in the Component Tree (left bottom tree) and will show you necessary information on the right side of the inspector’s window.
The target UI control will be highlighted.

Inspecting UI Controls

JVM Inspector also provides a functionality for quick element information search using hovering over an element (aka inspecting):

Click the Inspect button in the bottom left panel.
Hover cursor over the UI element and press ESC button. When you hover over an element, it is highlighted with a red border.
As a result:
- the target element is highlighted in the Component Tree;
- Unique descriptor and Component properties are displayed on the right panel of JVM Inspector.

When you have found the target component and tested the descriptor of this component, copy the descriptor to the machine configuration.

$("[CLASS:JCheckBox; Text:Telecom]").click()

Powerful Search selectors, using either XPath or CSS, to find specific elements:

CSS

$(".ParentC .Child3 .JButton").click()

XPath

$(byXpath("//element[@class=Child3]/element[@class=JButton")).click()

Combination of different selectors:

$("[XPATH: //element[@class=Child3]/element[@class=JButton]; NAME: Button Name; TEXT: Text Control]").click()

Use the XML export feature to obtain the XML file for building the selectors

Exporting Component Tree to XML

You can also get the whole UI component tree and manipulate it in your text editor or IDE with the XML export feature.

When you have a file with XML export, you can try building CSS, XPath, or RegExp selectors to uniquely locate an element in your RPA script, for example, [XPATH: //element[@class=Child3]/element[@class=JButton]; NAME: ButtonName; TEXT: TextControl].

There are two options: open XML code in a new window or save it to a new file.

Troubleshooting

JVM Inspector

Wrong Java version

In case you are unable to setup RPA Package Java as default Java for IE (jdk/jre.exe installer can do that anyway), you'll need to load Java with build version which is not higher than the RPA Package one.

Having RPA Package Java version 8u92 and IE pointed to 8u101, JVM-Inspector is unable to connect to 8u101 JVM and display elements selectors. Even when selector was found through different Machine, RPA Agent is unable to find it.

Installing 8u91 and pointing to IE fixes that issue: JVM-Inspector starts working and Selectors are executed successfully.

Robot cannot find element and JVM Inspector not seeing other processes

Sometimes after Windows security updates your RPA agent stops finding elements and JVMInspector isn't detecting any java processes for inspection, in the command window instead of

...inspector.controller.JvmInspectorControllerImpl.updateVirtualMachineTree[:73] - Updating JVM Tree...
...common.utils.VirtualMachineConnector$VMConnection.connect[:140] - Using jvmAgentPath =     c:\rpa\rpa-grid\dependency\rpa-jvmagent.jar     
...inspector.controller.JvmInspectorControllerImpl.connectToJvmAndUpdateVmTree[:236] - Successfully connected     to: 23652

there are only updating JVM tree attempts:

bad
...inspector.controller.JvmInspectorControllerImpl.updateVirtualMachineTree[:73] - Updating JVM Tree...
...inspector.controller.JvmInspectorControllerImpl.updateVirtualMachineTree[:73] - Updating JVM Tree...
...inspector.controller.JvmInspectorControllerImpl.updateVirtualMachineTree[:73] - Updating JVM Tree...

You may start troubleshooting by looking at the c:\rpa\java\bin\jconsole.exe.

When you see that JConsole detects external processes, but can't get any information off them too, and the warning message: The management agent is not enabled for this process.

It means, that JConsole can't access PID files created by a Java process.

To resolve the issues:

Close all Java programs.
Check where %TEMP% and %TMP% environment variables are pointed.
Clean your TEMP folder, most likely in C:/Users/{username}/AppData/Local/Temp.
Run settings-rdp.reg from the RPA setup folder.
Re-log in into your PC.
Run all the Java programs, test with jconsole, and then use JVM Inspector.
Check if C:/Users/{username}/AppData/Local/Temp/hsperfdata_{username} has files corresponding to Java process IDs.

AutoIt

User and System Java versions

Due to automatic java update, the AutoIt driver may suddenly stop picking up controls of a java application or applet. If this is the case you're facing, please make sure the java version used by the RPA Agent and the java application you are working with are started by the same java and their corresponding JVMs will be of the same version.

The easiest way to check which Java is used by default on a computer - is to use Java Control Panel. From there, navigate to Java tab, then click View button, then make sure that only one, and identical, JRE is selected on the System and User tabs.

When the path to system Java is determined, the same path has to be set for the JAVA_HOME variable in your start-node{N}.bat file:

...
set JAVA_HOME=E:/work/Java/jdk1.8.0_91
cd %~dp0 
%JAVA_HOME%\java -Dfile.encoding=UTF-8 -Dlogback.configurationFile=logback-node0.xml -cp      %JAVA_HOME%\lib\tools.jar;*;dependency\* com.workfusion.grid.CustomGridLauncher -role node -nodeConfig .\node0.json
...

To troubleshoot a Java applet, it may also help to enable console logging and tracing:

Example of working with MS Windows SysTreeView32 of Computer Management window

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy" charset="UTF-8">
    <selenium-flow>
        <selenium name="autoitDriver" browser="autoit" close-on-completion="false">
        <capability name="SEARCH_ALL_WINDOWS" value="true"/>
            <var-def name="scr1"><![CDATA[
                Local $hWnd = WinWait("[CLASS:MMCMainFrame; TITLE:Computer Management]", "", 10)
                If $hWnd Then
                    ControlTreeView($hWnd, "", "SysTreeView321", "Collapse","Computer Management (Local)|System Tools")
                    Local $nodeText = ControlTreeView($hWnd, "", "SysTreeView321", "GetText","#0|#0")
                    MsgBox(4096, "AutoIt Example", "tree:" & $nodeText)
                    ClipPut($nodeText)
                    Exit
                EndIf
            ]]></var-def>
            <script><![CDATA[
                try{
                    driver().switchTo().window('[CLASS:MMCMainFrame; TITLE:Computer Management]'); 
                    driver().executeScript(scr1.toString()); 
                    String clipboardText = driver().findElement(By.cssSelector("")).getAttribute("ACTION_GET_CLIPBOARD_TEXT");
                    println '______'+clipboardText 
                } catch(Exception e) {
                    println e
                }
            ]]></script>
        </selenium>
    </selenium-flow>
<export include-original-data="false"/>
</config>

WorkFusion Documentation

8.5

Set up environment

Explore project architecture

Locate UI elements and windows

Need of third party tools in RPA

AutoIt window info tool

JVM Inspector

Working with Component Tree

Using Search for Testing Descriptors

Inspecting UI Controls

Exporting Component Tree to XML

Troubleshooting

JVM Inspector

Wrong Java version

Robot cannot find element and JVM Inspector not seeing other processes

AutoIt

User and System Java versions