I'm new to using this platform, I'm sorry if the way I asked was wrong. i want to get data from twitter(X) using selenium, but there is something wrong,and I don't understand. i wanted to use edge driver because I had a google driver version error
the output I get when I run the code AttributeError: 'SeleniumClient' object has no attribute 'Edge_options'
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import pandas as pd
class SeleniumClient(object):
def __init__(self):
#Initialization method.
self.Edge = webdriver.EdgeOptions()
self.Edge_options('--headless')
self.Edge_options.add_argument('--no-sandbox')
self.Edge_options.add_argument('--disable-setuid-sandbox')
# you need to provide the path of chromdriver in your system
self.browser = webdriver.Edge(executable_path=r'C:\Users\Trinity\edgedriver\msedgedriver.exe', options=self.Edge_options)
self.base_url = 'https://twitter.com/search?q='
def get_tweets(self, query):
'''
Function to fetch tweets.
'''
try:
self.browser.get(self.base_url+query)
time.sleep(2)
body = self.browser.find_element_by_tag_name('body')
for _ in range(3000):
body.send_keys(Keys.PAGE_DOWN)
time.sleep(0.3)
timeline = self.browser.find_element_by_id('timeline')
tweet_nodes = timeline.find_elements_by_css_selector('.tweet-text')
return pd.DataFrame({'tweets': [tweet_node.text for tweet_node in tweet_nodes]})
except:
print("Selenium - An error occured while fetching tweets.")
selenium_client = SeleniumClient()
tweets_df = selenium_client.get_tweets('AI and Deep learning'
when I ran the code, I thought I could get data from twitter(tweet scrapping)
Selenium expects the options to be set the following way. This is only applicable for Selenium 4 upwards.
For more details see: https://www.selenium.dev/documentation/webdriver/browsers/edge/