Automate dynamic web elements

Most of sites are dynamic and change the page content by JS scripts while a user interacts with a page.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

   <robotics-flow>
       <robot name="roboticsDriver" driver="chrome" close-on-completion="true" start-in-private="true">
           <script><![CDATA[                                
               timeouts(15 * 1000);
               open("https://www.alibaba.com");
               $(byXpath("//input[@name='SearchText']")).val('sword');
               $(byXpath("//input[@value='Search']")).click();
               result = []  
               5.times { page ->  
                    elem = $(byXpath("//div[contains(@class,'ui2-pagination-pages')]/a[@class='next']")).click()
               }          
           ]]></script>
       </robot>
   </robotics-flow>

   <export include-original-data="true"></export>
</config>

In the example above, you access http://alibaba.com, enter a product name, and try to swap the page in the loop. If you run this Bot Task, it finishes with the exception: "unknown error: Element <a href="javascript:void(0)" className="next" data-role="next">...</a> is not clickable at point (1320, 994). Other element would receive the click..."

The exception occurs as the bot produces actions like a human. It moves the cursor to the element and sends a signal that the mouse button is pressed. This variant works in most cases, but not for this site. While the mouse cursor moves to the Next Page button, the site loads more elements, and the button changes its coordinates, so a click is sent to another UI element.

There are two main variants how to avoid errors: by using the hover() method or JavaScriptExecutor.

Click by JS

You can write a JS script that presses the button. That method is preferable since the button is pressed without scrolling and moving the cursor.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

   <robotics-flow>
       <robot name="roboticsDriver" driver="chrome" close-on-completion="true" start-in-private="false">
           <script><![CDATA[                                
               timeouts(40 * 1000);
               open("https://www.alibaba.com");
               $(byXpath("//input[@name='SearchText']")).val('sword');
               $(byXpath("//input[@value='Search']")).click();
               result = []  
               5.times { page ->  
                    element = $(byXpath("//div[contains(@class,'ui2-pagination-pages')]/a[@class='next']"));
                    JavascriptExecutor js = (JavascriptExecutor)driver();
                    js.executeScript("arguments[0].click();", element);
                    sleep(2000); // isn't really necessary, but helps to you to check, that pages are changing,
                                 // without sleeping they will change to fast for naked eye.
               }          
           ]]></script>
       </robot>
   </robotics-flow>

   <export include-original-data="true"></export>
</config>

Use hover() method

You can use the hover() method for loading extra elements before you click the button.

Move the cursor to the element before clicking:

    elem = $(byXpath("//div[contains(@class,'ui2-pagination-pages')]/a[@class='next']")).hover()
    elem.click()

However, it doesn't help as the bot needs more time to update its cash. Add sleep or wait to the code:

    elem = $(byXpath("//div[contains(@class,'ui2-pagination-pages')]/a[@class='next']")).hover()
    sleep(1000)
    elem.click()

It still generates an exception: "stale element reference: element is not attached to the page document."

The error occurs because the page was changed, but the bot tries to access the element presented on the previous page and needs some time to reevaluate XPath. Add another sleep.

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns="http://web-harvest.sourceforge.net/schema/1.0/config" scriptlang="groovy">

   <robotics-flow>
       <robot name="roboticsDriver" driver="chrome" close-on-completion="true" start-in-private="false">
           <script><![CDATA[                                
               timeouts(40 * 1000);
               open("https://www.alibaba.com");
               $(byXpath("//input[@name='SearchText']")).val('sword');
               $(byXpath("//input[@value='Search']")).click();
               result = []  
               5.times { page ->  
                    elem = $(byXpath("//div[contains(@class,'ui2-pagination-pages')]/a[@class='next']")).hover()
                    sleep(1000)    //elem = $(byXpath("//div[contains(@class,'ui2-pagination-gotopage')]/input[@class='ui2-pagination-goto']")).val(page)
                    elem.click()
                    sleep(3000)
               }          
           ]]></script>
       </robot>
   </robotics-flow>

   <export include-original-data="true"></export>
</config>