Fastest and Best way to store and access complex JSON objects on Redis

Question

I've been using ReJSON(json.set) for storing complex JSON on my redis server. For ex:

{'2018-02-01' : {'cid_1':{ 'city_1: {'mid_1: {'user_data : ...},{'merchant_data': ...},{'item_data':...}...}...}...}

Accessing one key at a time is blazingly fast. But accessing months of data, and adding it, takes quite an appreciable amount of time.

Is there another better way to store/access these complex json structures:

1) So if I just need user_data, not having to retrieve all the other data and then filter out on the rest, like:

dict_a = rj.jsonget(self.start_date, rejson.Path.rootPath())
dict_a = dict_a[self.cid][self.city][self.merchant]['User_data']

After testing on time, i see that 99% of it is spent getting and calculating the data. So based upon that, do you think my code needs more optimization ?

def calculate_total(self,T):
        delta = self.delta()
        for i in range(delta):
            try:
                dict_a = rj.jsonget(self.start_date, rejson.Path.rootPath())
                if T == 1:
                    dict_a = dict_a[self.cid][self.city][self.merchant]['Merchant_data']
                elif T == 2:
                    dict_a = dict_a[self.cid][self.city][self.merchant]['User_data']
                elif T == 3:
                    dict_a = dict_a[self.cid][self.city][self.merchant]['Item_data']
                break
            except KeyError:
                self.start_date = str((datetime.strptime(self.start_date, '%Y-%m-%d') + timedelta(days=i)).date())
            else:
                return ('Error 404- No Data found for %s, in %s on %s'%(self.cid,self.city,start_date))

        for i in range(delta):
            new_date = str((datetime.strptime(self.start_date, '%Y-%m-%d') + timedelta(days=i+1)).date())
            try:
                dict_b = rj.jsonget(new_date, rejson.Path.rootPath())
                if T == 1:
                    dict_b = dict_b[self.cid][self.city][self.merchant]['Merchant_data']
                elif T == 2:
                    dict_b = dict_b[self.cid][self.city][self.merchant]['User_data']
                elif T == 3:
                    dict_b = dict_b[self.cid][self.city][self.merchant]['Item_data']
                else:
                    dict_b = rj.jsonget(new_date, rejson.Path.rootPath())
                dict_a = merge_dict(dict_a,dict_b)
            except KeyError:
                pass
        return (dict_a)

def merge_dict(dictA, dictB):
    new_dict = {}
    common_keys = set([key for key in dictA if key in dictB] + [key for key in dictB if key in dictA])
    for k, v in dictA.items():
        #add unique k of dictA
        if k not in common_keys:
            new_dict[k] = v

        else:
            #add inner keys if they are not containing other dicts 
            if type(v) is not dict:
                if k in dictB:
                    new_dict[k] = v + dictB[k]
            else:
                #recursively merge the inner dicts
                new_dict[k] =  merge_dict(dictA[k], dictB[k])

    #add unique k of dictB
    for k, v in dictB.items():
        if k not in common_keys:
            new_dict[k] = v

    return new_dict

Answer 1

Instead of storing this complex json in redis, It needs to be stored as piece by piece using the redis data structure. While storing in this way keep in your mind the query which need to be executed at the time of retrieve data. This will bring you to the properly use of different redis data structure to minimise the query execution time.