Deadline exceeded while waiting for HTTP response, Python, Google App Engine

Question

I am using a cron job to download a web page and save it, using Google App Engine.

After 5 seconds I get a Dealine exceeded error.

How can I avoid the error. Searching around this site, I can extend the time limit for urlfetch. But it isn't urlfetch that is causing issues. It is the fact that I can't run the task for over 5 seconds.

For example, I have tried this fix, but it only works if I am running the page myself, not via a cron job:

HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline

Answer 1

Cron jobs can run for a maximum of 10 minutes, but that doesn't mean the URLFetch can't timeout before then. The default timeout for URLFetch is exactly 5 seconds but you can raise this.

"You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests. When using the URLConnection interface, the service uses the connection timeout (setConnectTimeout()) plus the read timeout (setReadTimeout()) as the deadline."

see: https://developers.google.com/appengine/docs/java/urlfetch/?csw=1#Requests

"A cron job will invoke a URL, using an HTTP GET request, at a given time of day. An HTTP request invoked by cron can run for up to 10 minutes, but is subject to the same limits as other HTTP requests."

see: https://developers.google.com/appengine/docs/python/config/cron

Furthermore check out this older stackoverflow question with very similar issue. If Google App Engine cron jobs have a 10 minute limit, then why do I get a DeadlineExceededError after the normal 30 seconds?