How to get all elements from javascript rendering page with Python and Requests-HTML

Question

I am learning web scraping and I installed requests-html. I want to scrap all "a" elements with class="screener-link-primary" from this page: finviz

This is my code:

from bs4 import BeautifulSoup
from requests_html import HTMLSession

url = 'https://finviz.com/screener.ashx?v=111&f=sec_technology'

session = HTMLSession()
response = session.get(url)
response.html.render()

soup = BeautifulSoup(response.html.html, "html.parser")
tickers = soup.find_all('a', class_ = "screener-link-primary")
print(tickers)

I can't get a single "a" element with "class=screener-link-primary". I printed the "response.html.html" and It gave me the source code of the page, but it's not the complete source code. So, I've been reading on multiple pages that the problem could be based on javascript rendering, that's why I'm using "response.html.render()" in order to simulate the web-browser behavior with the javascript effect, but it doesn't work.

Please help! PD: Thanks in advance.

enter image description here

Answer 1

i also do web scraping since 8 months ,Dont Know About requests_html but i'll use mostly selenium or ascraper. the output you want can be get through the following code easily using ascraper.

from ascraper.ascraper import scrapa
                  # added . before classname as if you know html css . means class
selector = {"css":".screener-link-primary"}
url = "https://finviz.com/screener.ashx?v=111&f=sec_technology"

s = scrapa()
s.CaptureData(
    url=url,
    selector=selector,
    mode="s",
    captureType="dynamic",
    filename="test1"

)

You can also prefer to the ascraper documentation. github :- https://github.com/abhaysakariya/ascraper

this module is build by myself to simplyfy the web scraping task to the users or programmer i hope may this help you and if not or found any bug or problem please tell me.