GCP Proto Datastore encode JsonProperty in base64

Question

I store a blob of Json in the datastore using JsonProperty. I don't know the structure of the json data.

I am using endpoints proto datastore in order to retrieve my data.

The probleme is the json property is encoded in base64 and I want a plain json object.

For the example, the json data will be:

{
    first: 1,
    second: 2
}

My code looks something like:

import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel


class Model(EndpointsModel):
    data = ndb.JsonProperty()


@endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):

    @Model.method(path='mymodel2', http_method='POST',
                  name='mymodel.insert')
    def MyModelInsert(self, my_model):
        my_model.data = {"first": 1, "second": 2}
        my_model.put()
        return my_model

    @Model.method(path='mymodel/{entityKey}',
                  http_method='GET',
                  name='mymodel.get')
    def getMyModel(self, model):
        print(model.data)
        return model


API = endpoints.api_server([DataEndpoint])

When I call the api for getting a model, I get:

POST /_ah/api/myapi/v1/mymodel2
{
    "data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}

where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}

And the print statement give me: {u'second': 2, u'first': 1}

So, in the method, I can explore the json blob data as a python dict. But, in the api call, the data is encoded in base64.

I expeted the api call to give me:

{
    'data': {
        'second': 2,
        'first': 1
     }
}

How can I get this result?

Answer 1

After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:

• It will show as plain JSON after the API call.
• You will be able to parse it again to a Python dict using eval.

I hope I understood correctly your issue, and this helps you with it.

import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel

class Sample(EndpointsModel):
  column1 = ndb.StringProperty()
  column2 = ndb.IntegerProperty()
  column3 = ndb.StringProperty()

@endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
  # URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
  @Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
  def MyModelInsert(self, my_model):
    dict={'first':1, 'second':2}
    dict_str=str(dict)

    my_model.column1="Year"
    my_model.column2=2018
    my_model.column3=dict_str
    my_model.put()
    return my_model

  # URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
  @Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
  def MyModelGet(self, my_model):
    if not my_model.from_datastore:
      raise endpoints.NotFoundException('MyModel not found.')

    dict=eval(my_model.column3)
    print("This is the Python dict recovered from a string: {}".format(dict))

    return my_model

application = endpoints.api_server([MyApi], restricted=False)

I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.

After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:

Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:

And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):

I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.

---

Update:

I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:

#Define the nested model (your JSON object)
class Structured(EndpointsModel):
  first = ndb.IntegerProperty()
  second = ndb.IntegerProperty()

#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
  column1 = ndb.StringProperty()
  column2 = ndb.IntegerProperty()
  column3 = ndb.StringProperty()
  column4 = ndb.StructuredProperty(Structured)

#Modify this endpoint definition to add a new property
@Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
  def MyModelGet(self, my_model):
    if not my_model.from_datastore:
      raise endpoints.NotFoundException('MyModel not found.')

    #Add the new nested property here
    dict=eval(my_model.column3)
    my_model.column4=dict
    print(json.dumps(my_model.column3))
    print("This is the Python dict recovered from a string: {}".format(dict))

    return my_model

With these changes, the response of the call to the endpoint looks like:

Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.

I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.