Edit

Surface-based Robotics driver

To automate console and core applications where there is no way to get a window or element locator, you can use WorkFusion's image-based (or surface-based) driver for automating desktop and web applications.

For faster image capturing and defining offsets, you can use the RPA Express Recorder for:

• Making screenshots
• Working with the media files panel
• Exporting code

Surface Selector: byImage()

The Robotics API has been extended with a byImage(String imageUrl, int offsetX, int offsetY) selector, which enables you to locate interface elements by their screenshots.

The bot performs click, hover, or other actions directly into the geometrical center of the screenshot (with an offset in pixels if defined):

• Offset X coordinate is positive from center to the right.
• Offset Y coordinate is positive from center to bottom.

Center click	Click with offset

The Robotics API example is as follows:

$(byImage("https://server-name/1478701332260-click.png")).doubleClick();

Images should be uploaded to a server and should be accessible through HTTP.

Surface Capability: imageSimilarityThreshold

For surface-based automation, it is possible to set the image similarity threshold capability, which can help solve complicated cases where images should strictly match (or alternatively be alike by 60%).

The imageSimilarityThreshold capability can take double values from 0.0 to 1.0.

The imageSimilarityThreshold syntax is as follows:

<robotics-flow>
        <robot name="driver" driver="desktop" close-on-completion="true">
             
            <capability name="imageSimilarityThreshold" value="0.6"/>
             
            <script><![CDATA[
                ...

You can also override this capability multiple times in the script section, but do not forget to import the org.sikuli.basics.Settings class.

To change image similarity inside a script, see below:

<robotics-flow>
    <robot name="driver" driver="desktop" close-on-completion="true">
 
        <script><![CDATA[
          import org.sikuli.basics.Settings;
 
          // ...
          // Set image similarity threshold to 0.5
           
          Settings.MinSimilarity = 0.5;
          $(byImage("${imagePath}/calculator2.png")).click();
 
          // ...
          // Set image similarity threshold to 0.9 which is a higher value
          // to exclude finding similar false images
 
          Settings.MinSimilarity = 0.9;
          $(byImage("${imagePath}/notepad.png", -10, 25)).click();
           
        ]]></script>
    </robot>
</robotics-flow>

Type without Window switching

When using the surface-based approach, you can add the following method if you want to type into any currently active window without explicitly switching to it:

• In the script section, add the enableTypeOnScreen() method.
• To disable this behavior, use the disableTypeOnScreen() method.

Enabling this typing option is not stable because random popup windows can appear while bot execution. We recommend using the window() method for each new window.

The typing on the screen example is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
 
    <robotics-flow>
        <robot driver="universal" close-on-completion="true" start-in-private="false">
            <capability name="SEARCH_ALL_WINDOWS" value="true" />
            <script><![CDATA[
                 
                // type into any currently active window without explicitly switching to it
                enableTypeOnScreen()               
                 
                open("notepad.exe")
                sendKeys("What is RPA?");
                pressEnter();
                 
                 
                openFirefox("https://www.wikipedia.org/");
                $(byXpath("//input[@id='searchInput']")).val("Robotic process automation").pressEnter();
                def wiki_text = $(byXpath("//div[@id='bodyContent']//p")).getText();
                 
                // explicitly switch to window before typing
                disableTypeOnScreen()
         
                window("[CLASS:Notepad]");
                sendKeys(wiki_text);
                pressCtrlA();
                pressBackSpace();
                 
            ]]></script>
        </robot>
    </robotics-flow>
    <export include-original-data="true"></export>
</config>

Examples

The next example shows clicking on Win10 calculator. Here the following images are used:

• calculator2.png

  

• menu.png

  

Win 10 calculator sample

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
 
    <robotics-flow>
        <robot name="driver" driver="desktop" close-on-completion="true">
            <capability name="imageSimilarityThreshold " value="0.2"/>
            <script><![CDATA[
                String imagePath = "https://rpa-grid.s3.amazonaws.com/integration-test/images-win-10/calculator";
 
                open("calc.exe");
                setOption("typeOnScreen", true);
 
                $(byImage("${imagePath}/calculator2.png", -10, 25)).click(); // 2
                $(byImage("${imagePath}/calculator2.png",  175, -75)).click(); // *
                $(byImage("${imagePath}/calculator2.png", -10, -30)).click(); // 5
                $(byImage("${imagePath}/calculator2.png",  230, 75)).click(); // =
                $(byImage("${imagePath}/menu.png")).click(); // show menu
 
                // copy result
                sys.defineVariable("calculationResult", selectAllTextAndCopy());
 
                // close
                pressAltF4();
            ]]></script>
        </robot>
    </robotics-flow>
 
    <export include-original-data="false">
        <single-column name="result" value="${calculationResult}"/>
    </export>
 
</config>

There is also an ability to get the size of the rectangle found on the screen by a bot and its location:

• getSize()
• getLocation()
• getRect()
  • getWidth()
  • getHeight()
  • getPoint()
  • getX()
  • getY()
• GetDimension()

Getting location and size sample

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">
 
    <robotics-flow>
        <robot name="driver" driver="desktop" close-on-completion="true">
            <capability name="imageSimilarityThreshold " value="0.3"/>
            <script><![CDATA[
                String imagePath = "https://rpa-grid.s3.amazonaws.com/integration-test/images-win-10/calculator";
 
                open("calc.exe");
                setOption("typeOnScreen", true);
 
                // execute coordinate-based methods
                sys.defineVariable("rectangle_width", $(byImage("${imagePath}/calculator2.png")).getRect().getWidth());
                sys.defineVariable("location", $(byImage("${imagePath}/calculator2.png")).getLocation());
                sys.defineVariable("size", $(byImage("${imagePath}/calculator2.png")).getSize());
 
                // close
                pressAltF4();
            ]]></script>
        </robot>
    </robotics-flow>
 
    <export include-original-data="false">
        <single-column name="rectangle_width" value="${rectangle_width}"/>
        <single-column name="location" value="${location}"/>
        <single-column name="size" value="${size}"/>
    </export>
 
</config>

Image-based RPA sample

def imagePath = "https://your-server/some-folder/";
  
$(byImage("${imagePath}/image1.png")).click();
$(byImage("${imagePath}/image1.png"), 40, -60).doubleClick();
$(byImage("${imagePath}/image1.png")).tripleClick();
$(byImage("${imagePath}/image1.png")).click(n);
$(byImage("${imagePath}/image1.png")).contextClick();
  
$(byImage("${imagePath}/image2.png")).hover();
 
$(byImage("${imagePath}/image3.png")).isExists();
$(byImage("${imagePath}/image4.png"), -25, 77).getLocation();
$(byImage("${imagePath}/image5.png"), 10, 77).getCoordinates();
 
actions()
   .dragAndDrop(
       $(byImage("${imagePath}/source-folder.png")),
       $(byImage("${imagePath}/target-folder.png")))
   .build().perform();
  
actions().dragAndDrop($(byImage("${imagePath}/source.png")), xOffset, yOffset).build().perform();
actions().clickAndHold($(byImage("${imagePath}/image.png"))).build().perform();
actions().release($(byImage("${imagePath}/image.png"))).build().perform();
 
actions().moveToElement($(byImage("${imagePath}/image.png"))).build().perform();