Gramex service Redis Cache throws an error sometimes "TypeError: can't pickle _thread.RLock objects"

Question

I am using the Gramex Redis Cache service but facing the below error sometimes which is happening randomly and it throws the error in gramex console, could you please help?

here is the error log from gramex console

Error log -

E 19-Jul 07:23:59 gramex:cache 8000 gramex.cache.open: <class 'gramex.services.rediscache.RedisCache'> cannot cache <tornado.template.Template object at 0x7f5cc43c1290>
Traceback (most recent call last):
  File "/home/star/conda/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/home/star/conda/lib/python3.7/site-packages/gramex/handlers/filehandler.py", line 188, in get
    yield self._get_path(self.root)
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 234, in wrapper
    yielded = ctx_run(next, result)
  File "/home/star/conda/lib/python3.7/site-packages/gramex/handlers/filehandler.py", line 244, in _get_path
    raise HTTPError(FORBIDDEN, f'{self.file} not allowed')
tornado.web.HTTPError: HTTP 403: Forbidden (/home/star/conda/lib/python3.7/site-packages/gramex/favicon.ico not allowed)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/home/star/conda/lib/python3.7/site-packages/gramex/cache.py", line 168, in open
    _cache[key] = cached
  File "/home/star/conda/lib/python3.7/site-packages/gramex/services/rediscache.py", line 58, in __setitem__
    value = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
TypeError: can't pickle _thread.RLock objects

Here is my gramex.yaml configuration set:

gramex.yaml config -

cache:
    memory:
      default: false
    redis:
      type: redis
      # path: $REDIS_CACHE_HOST:$REDIS_CACHE_PORT:$REDIS_CACHE_DB
      path: $REDIS_CACHE_HOST:$REDIS_CACHE_PORT:$REDIS_CACHE_DB:password=$REDIS_CACHE_PASSWORD
      size: 0 #GB cache
      default: true


app:
  session:
    type: redis         # Persistent multi-instance data store
    path: $REDIS_HOST:$REDIS_PORT:0:password=$REDIS_PASSWORD     # Redis server
    expiry: 10          # Session cookies expiry in days
    purge: 86400        # Delete old sessions periodically (in seconds)
    domain: .mydomain.com


url:
  show_daypart_data:
    pattern: /$YAMLURL/show_daypart_data
    handler: FormHandler
    kwargs:
      cors: true
      methods: $http_methods
      headers:
        $request_headers
      frm_30:
        url: $BigQ_CONN
        credentials_path: $CREDENTIAL_PATH
        state: my_utilities.cache_query(handler, '30 mi')
        queryfunction: star.get_show_daypart_query(handler, 'frm_period')
      modify: star.modify_show_daypart_data(handler, data)
      default:
        _limit: 450000
      error: *API_ERROR

Here is the "state" python function -

def cache_query(handler, table):
    '''Runs a cache validation query'''
    args = handler.argparse(
        src={'default': 'any'},
        view={'default': 'net'})
    state = ''
    for tbl in table.split('+'):
        table_name = rds_tables[args.src][tbl]
        query = f"""SELECT CONCAT(year, '-', week) as week FROM {table_name}
                        ORDER BY year DESC, week DESC limit 1"""
        try:
            val = REDIS_CACHE[str(query)]
            if not val:
                df = gramex.cache.query(query, db_engine)
                val = df.week.iloc[0]
                REDIS_CACHE[str(query)] = val
            state = '-' + val
        except Exception as error:
            app_log.error(f'Cache query failed ==> {error}')
            pass
    return state

Answer 1

This is an error in Gramex as of Jul 2022 (Release 1.81). It happens because

• The Gramex BaseHandler loads an error template with gramex.cache.open
• But the Redis Cache does not support caching such objects

So the custom error does not get displayed, and instead, it reports the TypeError above.

Short of avoiding the Redis Cache, there's no workaround for this right now, unfortunately. We plan to fix this in Gramex.