Having trouble using Beautiful Soup's 'Next Sibling' to extract some information

Question

On Auction websites, there is a clock counting down the time remaining. I am trying to extract that piece of information (among others) to print to a csv file.

For example, I am trying to take the value after 'Time Left:' on this site: https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx

I have tried 3 different options, without any success

1)

time = ''
    try:
        time = soup.find(id='tzcd').text.replace('Time Left:','')
        #print("Time: ",time)
    except Exception as e:
        print(e)

    time = ''
    try:
        time = soup.find(id='tzcd').text
        #print("Time: ",time)
    except:
        pass

3

time = ''
  try:
      time = soup.find('div', id="BiddingTimeSection").find_next_sibling("div").text
      #print("Time: ",time)
  except:
      pass

I am a new user of Python and don't know if it's because of the date/time structure of the pull or because of something else inherently flawed with my code.

Any help would be greatly appreciated!

Answer 1

That information is being pulled into page via a Javascript XHR call. You can see that by inspecting Network tab in browser's Dev tools. The following code will get you the time left in seconds:

import requests

s = requests.Session()

header = {'X-AjaxPro-Method': 'GetTimerText'}
payload = '{"inventoryId":271177}'
r = s.get('https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx')
s.headers.update(header)
r = s.post('https://auctionofchampions.com/ajaxpro/LotDetail,App_Web_lotdetail.aspx.cdcab7d2.1voto_yr.ashx', data=payload)
print(r.json()['value']['timeLeft'])

Response:

792309

792309 seconds are a bit over 9 days. There are easy ways to return them in days/hours/minutes, if you want.