404 when trying to create backup in a Google App Engine project

Question

Setup

I encountered the 404 problem after following the unaccepted answers of the question AppEngine datastore - backup programatically

I have enabled the Datastore Admin, as suggested by one of the answer provider. I can manually trigger a datastore backup in Google App Engine console and the backup runs without any failure.

The code in this question lives in a module called 'app'. Not 'default'.

The 404 Problem

This is the cron job in cron.yaml.

cron:
- description: Regular backup
  url: /_backup/fullbackup
  schedule: every 24 hours

The handler of url will put a backup task in a queue, which in turn make a call to

_ah/datastore_admin/backup.create?
gs_bucket_name=%2Fgs%2Ftest.appspot.com%2F21-06-2015&kind=Test&kind=TestContent
&kind=TestDocument&filesystem=gs

(I replaced my app id with 'test' here)

This shows a 404 error in the log.

If I use the above url with my app host name in a brower (i.e. https://test.appspot.com/_ah/datastore_admin/backup.create? gs_bucket_name=%2Fgs%2Ftest.appspot.com%2F21-06-2015&kind=Test&kind=TestContent &kind=TestDocument&filesystem=gs), I get a 404 too.

Here is the relevant code in the handler of the route /_backup/fullbackup

    task = taskqueue.add(
        url='/_ah/datastore_admin/backup.create',
        method='GET',
        target='ah-builtin-python-bundle',
        params={
            'filesystem': 'gs',
            'gs_bucket_name': self.get_bucket_name(),
            'kind': (
                'Test',
                'TestContent',
                'TestDocument'
            )
        }
    )

Questions:

• What is the cause of the issue?
• Do I need a queue name in the taskqueue.add part of python code?
• In my cron.yaml, do I need to set target to ah-builtin-python-bundle?

EDIT

The datastore-admin built-in has been enabled, as seen in this screenshot.

And there is no dispatch.yaml

Answer 1

It is related to a queue configuration issue.

A 'default' queue definition was present in the app.yaml prior to the implementation of regular backup. The backup tasks are, as a result, not reaching the target 'ah-builtin-python-bundle'

If I define a new queue

- name: data-backup
  rate: 1/s
  target: ah-builtin-python-bundle

Then use this code to insert task,

    task = taskqueue.add(
        url='/_ah/datastore_admin/backup.create',
        method='GET',
        queue_name="data-backup",
        params={
            'filesystem': 'gs',
            'gs_bucket_name': self.get_bucket_name(),
            'kind': kinds_list  # A list of ndb model classes I want to backup
        }
    )

Then Google app engine can create backup set

Answer 2

Have you enabled the Datastore Admin? You'll need to have done this to allow the module ah-builtin-python-bundle to exist, which is a special module "deployed" to your app when you activate the Datastore admin, which really is responsible for responding to requests to /_ah/datastore_admin and spawning MapReduce jobs that read from Datastore and produce the backup files in Cloud Storage (or wherever else you send them to).

Also, another possibility is that you've used test.appspot.com hard-coded into your app. Do you own that app id, "test"? From the screenshot of the error you see in the browser, it appears as though you're attempting to back-up to the bucket "test.appspot.com", which would be the default bucket for the app with app id "test". However, in the logs screenshot you show, it also attempts to back-up to the "example.appspot.com" bucket. Ensure that your app owns these bucket.

Another possibility is that the module which is handling the request isn't ah-builtin-python-bundle, but rather another one. This might happen, even if you specify a different target in the task add method, if you have a dispatch rule which is re-routing the request.