Stay logged in using MechanicalSoup

Question

I am trying to build a website scraper to pull URLs from a page. The scraper works fine when I am on the normal website but I want to login so that I can use the filter functions on the website. I use MechanicalSoup to login and the login is successful but when I move to the page using requests, I am no longer logged in. Please help!

Here is my code:

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup
import mechanicalsoup

URL = "https://www.morphmarket.com/us/search?q=&sex=&maturity=0&cat=31&min_genes=0&max_genes=9&traits=&neg_traits=&min_price=0&max_price=1000000&cur=USD&sort=nfs&epoch=2&store=&country=&export=&layout=grid&page=1"
LOGIN = "_______"
PASSWORD = "_______"

browser = mechanicalsoup.StatefulBrowser()
browser.open(URL)
browser.select_form('form[action="/accounts/login/"]')
browser.get_current_form().print_summary()
browser["login"] = LOGIN
browser["password"] = PASSWORD
response = browser.submit_selected()
print(browser.get_url())

snake_urls = []
for i in range(1, 3):
    number = str(i)
    my_url = 'https://www.morphmarket.com/us/search?q=&sex=&maturity=0&cat=31&min_genes=0&max_genes=9&traits=&neg_traits=&min_price=0&max_price=1000000&cur=USD&sort=nfs&epoch=2&store=&country=&export=&layout=grid&page=' + number
    uClient = uReq(my_url)
    page_html = uClient.read()
    uClient.close()
    page_soup = soup(page_html, "html.parser")
    containers = page_soup.findAll("div",{"class":"col-md-3 col-sm-4 col-ms-4 col-xs-6 item-col move-up"})

and then I go on to preform my scraping...

Answer 1

I think you start with mechanical soup to connect, but after you use an new request to open you link. If you use mechanicalsoup you should not use either request or beautifulsoup, because mechanical soup isalready based on these librairies. So you are not connected using a new request connection ! So call directly browser.open(my_url), and use beautifulsoup through browser object.