When I use requests.get(URL, headers), my custom headers go through fine - but when I use:
s = requests.Session()
r = s.get(url, headers)
I check the headers by both r.headers and s.headers to see what is being delivered or sent, and I can't seem to verify 100% that my custom headers are being sent through and overriding the requests.Session() generic headers.
I'm wondering how to verify this, and if I'm doing this the right way.
UPDATE: Someone responded outside of Stack Overflow that it is r.request.headers to check headers sent to the webserver.
Based on the articles I've read online, I'd like to start using requests.Session() objects for cookies, specifically. I don't want to hard code any cookies. On that note, speaking of cookies --- shouldn't the webserver I'm trying to scrape set the cookies when you send a get request. I'm not sure why I would need to send the cookies myself. For example, if you read cookie policies on websites, it sounds as if the website assigns the cookies and the user doesn't need to send cookies to the website. The website sends cookies back to the user. I might be misunderstanding. Please explain.
UPDATE: So, from my understanding at this point is that the only reason why I would want to create a session object is for websites that require login and persistent cookies to maintain the status as logged in ... And, in general - worrying about cookies sent in a get request is not necessary.