Changing the referer URL in python requests

Question

How do I change the referer if I'm using the requests library to make a GET request to a web page. I went through the entire manual but couldn't find it.

Answer 1

According to http://docs.python-requests.org/en/latest/user/advanced/#session-objects , you should be able to do:

s = requests.Session()
s.headers.update({'referer': my_referer})
s.get(url)

Or just:

requests.get(url, headers={'referer': my_referer})

Your headers dict will be merged with the default/session headers. From the docs:

Any dictionaries that you pass to a request method will be merged with the session-level values that are set. The method-level parameters override session parameters.

Answer 2

here we are rotating the user_agent with referer

user_agent_list = [
   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
   "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36",
   "Mozilla/5.0 (iPad; CPU OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/104.0.5112.99 Mobile/15E148 Safari/604.1"
]
reffer_list=[
   'https://stackoverflow.com/',
   'https://twitter.com/',
   'https://www.google.co.in/',
   'https://gem.gov.in/'
]
headers = {
   'Connection': 'keep-alive',
   'Cache-Control': 'max-age=0',
   'Upgrade-Insecure-Requests': '1',
   'User-Agent': random.choice(user_agent_list),
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
   'Accept-Encoding': 'gzip, deflate',
   'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8',
   'referer': random.choice(reffer_list)
}