How can I make my wesbcraper type something and select from a dropdown

Question

So I want to scrape all the Chiropractors from this website https://chiropractic.ca/find-a-chiro/alberta/

I use a tool called Webscraper, which is very good, allow me to do pagination and further select the data of each chiropractor to store it in a sheet.

The problem here is that for the chiropractors to appear you must paste a city on the search tab, as well as select a radius.

So I need to know how can I instruct my webscraper to first type in the city and choose a radius from the dropdown.

I am not sure if webscraper.io can do this, but any help I would really really appreciate.

Haven't found no solution

Answer 1

To make your web scraper type something into a text input field and select from a dropdown on a web page, you can use Selenium, a powerful tool for automating web browsers. Here's a basic example of how you can achieve this using Ruby and Selenium:

require 'selenium-webdriver'

# Start the web browser
driver = Selenium::WebDriver.for :chrome

# Open the web page
driver.get("https://chiropractic.ca/find-a-chiro/alberta/")

# Find the text input field and type something into it
input_field = driver.find_element(id: "addressInput_7")
input_field.send_keys("Your text here")

# Find the dropdown menu and select an option by value
dropdown = driver.find_element(id: "radiusSelect_7")
select = Selenium::WebDriver::Support::Select.new(dropdown)
select.select_by(:value, "option_value")

# Close the browser
driver.quit