Fetching 100 Results at a Time in Google App Engine

Question

I was hoping someone could explain to me how to use offsets or cursors in App Engine. I'm using gcloud to remote access entities for a huge data migration, and would like to grab data in batches of 100.

I'm guessing there is a very simple way to do this, but the documentation doesn't dive into cursors all too much. Here is what I have so far:

client = datastore.Client(dataset_id=projectID)

# retrieve 100 Articles
query = client.query(kind='Article').fetch(100)

for article in query:
  print article

How could I mark the end of that batch of 100 and then move into the next one? Thanks so much!

Edit:

I should mention that I do not have access to the app engine environment, which is why I'm a bit lost at the moment... :(

Answer 1

I don't have any experience with gcloud, but I don't think this should be too different.

When you query, you will use fetch_page instead of the fetch function. The fetch_page function returns three things (results, cursor, more). The cursor is your bookmark for the query, and more is true if there are probably more results.

Once you've handled your 100 entities, you can pass on the cursor in urlsafe form to your request handler's URI, where you will continue the process starting at the new cursor.

from google.appengine.datastore.datastore_query import Cursor

class printArticles(webapp2.RequestHandler):
    def post(self):

        query = Client.query()

        #Retrieve the cursor. 
        curs = Cursor(urlsafe=self.request.get('cursor'))

        #fetch_page returns three things
        articles, next_curs, more = query.fetch_page(100, start_cursor=curs)

        #do whatever you need to do
        for article in articles:
            print article

        #If there are more results to fetch
        if more == True and next_curs is not None:

            #then pass along the cursor
            self.redirect("/job_URI?cursor=" + next_curs.urlsafe())