I can't create a Google Cloud Storage file working on "dev_appserver.py."

Question

I caught an error as below.

INFO     ~ module.py:639] default: "HEAD /_ah/gcs/app_default_bucket/multibytes.txt HTTP/1.1" 404 -
ERROR    ~ gcs.py:99] Expect status [200] from Google Storage. But got status 404.
Path: '/app_default_bucket/multibytes.txt'.
Request headers: None.
Response headers: {'date': 'Mon, 07 Jul 2014 12:59:44 GMT', 'server': 'Development/2.0', 'connection': 'close'}.
Body: ''.
Extra info: None.
Traceback (most recent call last):
  File "/gcs.py", line 97, in status
        stat = gcs.stat("/%s/%s" % (b,nm))
  File "/cloudstorage/cloudstorage_api.py", line 142, in stat
        body=content)
  File "/cloudstorage/errors.py", line 132, in check_status
        raise NotFoundError(msg)
NotFoundError: Expect status [200] from Google Storage. But got status 404.
Path: '/app_default_bucket/multibytes.txt'.
Request headers: None.
Response headers: {'date': 'Mon, 07 Jul 2014 12:59:44 GMT', 'server': 'Development/2.0', 'connection': 'close'}.
Body: ''.
Extra info: None.

this is my custom GCS cliet class for example.

# encoding: utf-8

import cloudstorage as gcs

class mycustomgcsclient:

  #...

  def create(self,name,data,**options):
    options['retry_params'] = gcs.RetryParams(backoff_factor=1.1)

    if not options.get('content_type'):
      options['content_type'] = 'octet-stream'

    if isintance(name,unicode):
      name = name.encode('utf-8')

    path = '/mybucketname/%s' % name
    try:
      with gcs.open(path,'w',**options) as f:
        f.write(data)
      return True
    except Exception as e:
      logging.exception(e)

    return False

if __name__=='__main__':
  data = 'some data ...¥n'

  filename = 'somedir/%s' % u'sample.txt'
  mycustomgcsclient().create(filename,data) # no error occured.

  filename = 'somedir/%s' % u'あいうえお.txt'
  mycustomgcsclient().create(filename,data) # error occured in this line.

I have caught an error above when only using a multibytes filename.

I have not caught any error when using a ascii's filename.

I am using a "GCS Client Library (Python)" given by on https://developers.google.com/appengine/docs/python/googlecloudstorageclient/download.

My dev_appserver.py's version is Development SDK 1.9.6, and this is working on MacOS X Marve..(? forgotten).

Is there some solutions?

Answer 1

I think you may need to call urllib.quote() on the name after encoding it to utf8.

Here is a modified version of the GCS Python demo (http://appengine-gcs-client.googlecode.com/svn/trunk/python/demo/main.py) that works properly using a multibyte filename:

# Copyright 2012 Google Inc. All Rights Reserved.
# encoding: utf-8

"""A modified version of the sample app that uses GCS client to operate on
   bucket and file.
"""

import logging
import os
import cloudstorage as gcs
import webapp2
import urllib

from google.appengine.api import app_identity

my_default_retry_params = gcs.RetryParams(initial_delay=0.2,
                                          max_delay=5.0,
                                          backoff_factor=2,
                                          max_retry_period=15)
gcs.set_default_retry_params(my_default_retry_params)


class MainPage(webapp2.RequestHandler):
  """Main page for GCS demo application."""

  def get(self):
    bucket_name = os.environ.get('BUCKET_NAME',
                                 app_identity.get_default_gcs_bucket_name())

    self.response.headers['Content-Type'] = 'text/plain'
    self.response.write('Demo GCS Application running from Version: '
                        + os.environ['CURRENT_VERSION_ID'] + '\n')
    self.response.write('Using bucket name: ' + bucket_name + '\n\n')

    bucket = '/' + bucket_name
    filename = bucket + '/' + urllib.quote(u'あいうえお.txt'.encode('utf8'))
    self.tmp_filenames_to_clean_up = []

    try:
      self.create_file(filename)
      self.response.write('\n\n')

      self.read_file(filename)
      self.response.write('\n\n')

    except Exception, e:
      logging.exception(e)
      self.delete_files()
      self.response.write('\n\nThere was an error running the demo! '
                          'Please check the logs for more details.\n')

    else:
      self.delete_files()
      self.response.write('\n\nThe demo ran successfully!\n')

  def create_file(self, filename):
    """Create a file.

    The retry_params specified in the open call will override the default
    retry params for this particular file handle.

    Args:
      filename: filename.
    """
    self.response.write('Creating file %s\n' %
                        urllib.unquote(filename).decode('utf-8'))

    write_retry_params = gcs.RetryParams(backoff_factor=1.1)
    gcs_file = gcs.open(filename,
                        'w',
                        content_type='text/plain',
                        options={'x-goog-meta-foo': 'foo',
                                 'x-goog-meta-bar': 'bar'},
                        retry_params=write_retry_params)
    gcs_file.write('some data ...¥n\n')
    gcs_file.close()
    self.tmp_filenames_to_clean_up.append(filename)


  def read_file(self, filename):
    self.response.write('File Content:\n')

    gcs_file = gcs.open(filename)
    self.response.write(gcs_file.readline())
    gcs_file.close()

  def delete_files(self):
    self.response.write('Deleting files...\n')
    for filename in self.tmp_filenames_to_clean_up:
      self.response.write('Deleting file %s\n' %
                          urllib.unquote(filename).decode('utf-8'))
      try:
        gcs.delete(filename)
      except gcs.NotFoundError:
        pass

app = webapp2.WSGIApplication([('/', MainPage)],
                              debug=True)