How to print fp in HTTPError?

Question

After seeing this error with my urlopen() function:

    Traceback (most recent call last):
  File "test_urlopen.py", line 47, in <module>
    response = request.urlopen(req, data, context=ctx)
  File "/lib64/python/lib/urllib/request.py", line 227, in urlopen
    return opener.open(url, data, timeout)
  File "/lib64/python/lib/urllib/request.py", line 541, in open
    response = meth(req, response)
  File "/lib64/python/lib/urllib/request.py", line 653, in http_response
    response = self.parent.error(
  File "/lib64/python/lib/urllib/request.py", line 580, in error
    return self._call_chain(*args)
  File "/lib64/python/lib/urllib/request.py", line 508, in _call_chain
    result = func(*args)
  File "/lib64/python/lib/urllib/request.py", line 663, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400:

I wrote a catch exception, but fp is empty when I printed it out. Any ideas?

try:
    response = request.urlopen(req, data, context=ctx)
except urllib.error.HTTPError as e:
    print ("the exception is %s " % str(e.fp())
    print ("the exception is %s " % str(e.fp.readlines()))
    exit(0)

Output:

the exception is <http.client.HTTPResponse object at 0xd1b9615640>
the exception is []

fp isn't empty in the HTTPError class

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp)))
        print (fp.readlines())
        raise HTTPError(req.full_url, code, msg, hdrs, fp)

Output:

"fp <http.client.HTTPResponse object at 0xd1b9615640>
[b'{"code":"6006","reason":"Failed to update".....}']

Thanks.

Answer 1

Perhaps I'm dense; but a few items that might help you.

1. You have an errant " in your print("str(fp)). (Thanks to @CrazyChucky for pointing that out. Was particularly stupid of me to miss that. Made more sense without the ")

2. In your code:

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp))
        print (fp.readlines())
        raise HTTPError(req.full_url, code, msg, hdrs, fp)

I noticed you do a print (fp.readlines()) and then raise the error. After that print (fp.readlines()), the filepointer will be at the end of the file. So any further fp.readlines() will return []

I'd suggest the following:

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp))
        print (fp.readlines())
        fp.seek(0)
        raise HTTPError(req.full_url, code, msg, hdrs, fp)

After noticing that the OP got an error with the fp.seek(0), and noticing that fp is a HTTPResponse object, I'm not sure it's even possible to do an analogous seek() action as the internal fp of HTTPResponse is a socket file which (iiuc), doesn't allow seeking.

With that said, and if someone smarter than me can correct me, the options are:

1. Remove the fp.readlines() altogether as that will set the fp pointer to the end and you can't reset it.

2. Copy the fp.readlines() items to a new file object (BytesIO()) and pass that to HTTPError.

i.e.

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp))
        olddata = fp.readlines()
        new_fp = io.BytesIO(olddata)
        new_fp.seek(0)
        raise HTTPError(req.full_url, code, msg, hdrs, new_fp)

A possible problem that I see is if HTTPError requires new_fp to be a HTTPResponse object. But since this is a RAISE line, the script/program would stop (assuming here..).

This is what I believe will give you what you want, though I'm doubtful it is 100% what you want. I apologize.