My script grabs the content of an rss page gets the urls in that page saves them to a list then it grabs the content of each url and it emails the contents of the page to me. Everything is working very well accept I can't send every link in the list. Typically about 22 links in the list. I don't want to combine the contents of multiple links into one email. If I don't add a timeout I get an over quota error like this
<class 'google.appengine.runtime.apiproxy_errors.OverQuotaError'>: The API call mail.Send() required more quota than is available.
After I added "time.sleep(9)" to slow it down it gives me this error.
<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
Here is my code.. Any thoughts?
size = len(my_tabletest)
a=2
while a < size:
url = my_tabletest[a].split('html</link>')[0] + "print"
url_hhhhhh = urlfetch.fetch(url)
my_story = url_hhhhhh.content
my_story = my_story.split('<div class="printstory">')[1]
my_story_subject = my_story.split('<h1>')[1]
my_story_subject = my_story_subject.split('</h1>')[0]
my_story = ''.join(BeautifulSoup(my_story).findAll(text=True))
message = mail.EmailMessage(sender="me<[email protected]>",
subject=my_story_subject)
message.to = "Jim <[email protected]>"
message.body = my_story
message.html = my_story_html
message.send()
time.sleep(9)
a=a+1
Welcome to Stack Overflow!
The task queue is built to solve this problem. You can leverage it with minimal change to your existing code using the deferred library:
Instead of calling
message.send()
, do something like this:This will create a batch of ad-hoc tasks that send your emails in the background, after your main request handler has returned. Some of these tasks will probably fail on the first try as your app hits short term quota limits for outbound mail. That's OK; failed tasks will back off and retry automatically until they succeed.
Edit: Oh, and take the
sleep
out of your code. =)Edit #2: You can speed things up further by moving the urlfetch into the task, so each task fetches one URL and then sends one email. Fetching 22 URLs in one request handler could be enough to cause timeouts, independent of sending mail.