10.2.5

Edit

Craigslist bot

The Craigslist Bot searches http://sfbay.craigslist.org and returns the top results to a spreadsheet.

Search parameters are configurable. The bot execution takes around 55 seconds for 10 items.

note

Constructing this script from scratch requires knowledge of XPath. See XPath Guide for more details.

Quick links

Access bot

To access the bot, do as follows:

  1. Download and extract the craigslist-bot folder to the rpae_project folder in your workspace.

  2. To configure the Excel file path, right-click listings.xlsx on the Media Files tab, select Properties > Resource > Location, and copy the resource location.

  3. Go to the Excel Open spreadsheet action and insert the copied value to the File location field.

  4. Navigate to your spreadsheet, highlight all the data from five columns, select Clear Contents, and save.

Script workflow

Configure variables

Before you start building your script, define all the needed Recorder Variables:

  1. Set up the following String variables:

    1. search: the default search value. You can search for "apple watch", "macbook pro", "honda civic", and so on.
    2. item_name: item name.
    3. item_price: item price.
    4. item_loc: item location.
    5. item_link: item link.
    6. error_catch: the default value is NULL.

  2. Set up the following List variables:

    1. list_search
    2. list_name
    3. list_price
    4. list_loc
    5. list_link

  3. Add a Table variable (item_table) and a Number variable (count).

Specify search parameters

Actions to be used

  1. Go to Actions Library and add Web > Open Website to Actions Flow.

    1. Enter "sfbay.craigslist.org" into the Site URL field.
    2. Specify your browser in Browser.

  2. Add Web > Web Element to identify a search bar for entering a search request. Drag the action inside the Open Website group.

    1. Select Mode > Set value.
    2. Select the search variable for Input.
    3. Specify the element's XPath: //*[@id="query"].

  3. Add Keyboard > Enter Keystrokes. To set up the key, click Change within Key combination and press Enter.

Extract search results

Actions to be used

  1. Go to Actions Library and add Loops > Repeat. Enter 10 (times) for Repeat the nested actions.

  2. To instruct your Bot which search results to extract, add expression values. Add Variables > Expression for the list_search variable inside the Loops group.

    1. Select list_search for Variable.
    2. Check Push (append).
    3. Select search as Value to append.

  3. Add Web > Web Element to get the item name.

    1. Select Mode > Get value.
    2. Specify the element's XPath: //*/li[${count}]/p/a[@class="result-title hdrlnk"].
    3. Select item_name for Output.

  4. Add Variables > Expression for the list_name variable.

    1. Select list_name for Variable.
    2. Check Push (append).
    3. Select item_name as Value to append.

  5. Add Exception Handling to ensure safe execution of your workflow.

    1. Within Try to complete, add Web > Web element to get the item price.
      1. Select Mode > Get value.
      2. Specify the element's XPath: //*/li[${count}]/p/span[@class="result-meta"]/span[@class="result-price"].
      3. Select item_price for Output.

    1. Add Variables > Expression.
      1. Select list_price for Variable.
      2. Check Push (append).
      3. Select item_price as Value to append.
    2. Within If an exception occurred, add Variables > Expression.
      1. Select list_price for Variable.
      2. Check Push (append).
      3. Select error_catch as Value to append.

  6. Add one more Exception Handling.

    1. Within Try to complete, add Web > Web element to get the item location.
      1. Select Mode > Get value.
      2. Enter //*/li[${count}]/p/span[@class="result-meta"]/span[@class="result-hood"] in the XPath field.
      3. Select item_loc for Output.
    2. Add Variables > Expression.
      1. Select list_loc for Variable.
      2. Check Push (append).
      3. Select item_loc as Value to append.
    3. Within If an exception occurred, add Variables > Expression.
      1. Select list_loc for Variable.
      2. Check Push (append).
      3. Select error_catch as Value to append.

  7. Add Web > Web element to get the item link.

    1. Select Mode > Get value.
    2. Specify the element's XPath: //*/li[${count}]/p/a[@class="result-title hdrlnk"].
    3. Check Use element attribute and enter href.
    4. Select item_link for Output.

  8. Add Variables > Expression for the list_link variable.

    1. Select list_link for Variable.
    2. Check Push (append).
    3. Select item_link as Value to append.

  9. Add Variables > Expression for the count variable.

    1. Select count for Variable.
    2. Enter ${count}+1 for Expression.

  10. To populate the list, add Variables > Expression for the item_table variable.

    1. Select item_table for Variable
    2. Check Push (append).
    3. Select Column for Append as.
    4. Select a variable for Value to append. The variables to be selected in the field are as follows:
      • list_search
      • list_name
      • list_price
      • list_loc
      • list_link
    caution

    As the expression value is added for five variables mentioned above, repeat Step 10 five times in the script, with each variable selected for Value to append.

Enter results into Excel file

Actions to be used

  1. Go to Actions Library and add Excel > Open spreadsheet.

    1. Specify the file path in the File location field: C:/Users/username/workfusion-workspace-2/craiglist/rpae_project/listings.xlsx.
    2. To save the document when all the values are inserted, check Save file after last action.

  2. Add Excel > Set Range.

    1. Select item_table for Input.
    2. Go to Options and set up the range: From cell # – A2; To cell # – E26.

  3. Add the Wait action and set up Pause script execution (for 100 milliseconds).

Play script

When you finish building your script, you should have the following order of actions in Actions Flow:

Click Play Recording to view the Craigslist performance.

← TranslationWeb scraping →