I want to login to a website and navigate to a specific page to scrape the data. I am planning on using scraping (not API at the moment) and for learning purpose I plan on doing it on my stackoverflow account to extract how my reputation score has changed over time and on which topic.
And, I am using google apps script as a programming language, also for the learning purpose.
I am using the below given code for login:
function stackLogin() {
var url = "https://stackoverflow.com/users/login?ssrc=head";
//var url = "https://stackoverflow.com/";
var payload = {
"email":"myLogin",
"password":"myPassword"
};
var opt = {
"payload":payload,
"method":"post",
"followRedirects": false
};
var response = UrlFetchApp.fetch(url, opt);
var sessionDetails = response.getAllHeaders()['Set-Cookie'];
var header = {
'Cookie': sessionDetails[1]
};
Logger.log(response.getResponseCode());
Logger.log(response);
}
When I use:
url = "https://stackoverflow.com/users/login?ssrc=head"
I get "response code = 302" but the "response html" is very short.
I also see that a new login ip appears on StackExchange OpenID
.
But, if I use:
url = "https://stackoverflow.com"
I get "response code = 200" and the "response html" is very long. I also see that a new login ip does not appear on StackExchange OpenID.
Question 1)
So, the login was only possible by url = "https://stackoverflow.com/users/login?ssrc=head", rite ??
Question 2)
If I am able to login I want to click "my profile avatar button"
which takes me to the "user profile/summary" page where I want to click "Reputation"

which takes me to "Reputation log page" where I have summary of all the reputation I have received by date and topics.

Now, I want to extract these reputation values as table.
I can scrape the data by inspecting the class values of these reputation logs if I can pull the HTML page.
The main problem I have though is : How do I click through different buttons (after successfully logging in) to get to that "Reputation log" page.
