Unexpected behaviour after memcached server restarts. How to configure/rectify it?

Question

I have a pool of persistent connections(Memcached clients). Data are being cached in the memcached server. If after restarting the memcached server, I try to get the cached data using the client from the pool, I m getting the below exception:

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Cancelled
    at net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1662)
    at net.spy.memcached.MemcachedClient$GetFuture.get(MemcachedClient.java:1708)
    at com.eos.gds.cache.CacheClient.get(CacheClient.java:49)

I get this exception only for the first time after the restart when I try to get the cached data. I did a lot of search. But unable to find the exact reason for this.

Answer 1

Spymemcached has a bunch of internal queues that operations are placed in before they are actually sent out to memcached. What is happening here is that you do an operation and then before that operation is sent over the wire or before a response is received from memcached, Spymemcached realizes that the connection has been lost. As a result Spymemcached cancels all operations in flight and then reestablishes the connection.

When you call get() on the Future then since the operation was cancelled by Spymemcached an exception is thrown. What I recommend doing here is catching all exceptions on every individual operation you do with Spymemcached and then, depending on the error, either retrying the operation of just forgetting about it. If it's a get for example and your cluster of memcached servers goes down then you can probably forget about it since the cache will be empty, but you will probably want to retry a set.

Answer 2

I ran into the exact same problem and fix it by handling the exception until success

while(true){
 try{
  memcacheclient.get(key);
  break;
 }
 catch(java.util.concurrent.CancellationException e ){
  log.info("cache cancelled");
 }
}

Answer 3

Run MemcachedClient.getStats() for each new client once and that will fix the cancelations issue.

Answer 4

I had the same issue. I am using Spymemcached client to connect with Memcache server.

I found .

There must be a connection issue.

Ref: https://github.com/couchbase/spymemcached/blob/master/src/main/java/net/spy/memcached/internal/OperationFuture.java

Answer 5

Been searching for days for a solution. Posting in case it helps someone else.

Our implementation of ServletContextListener was getting a new MemcachedClient(...) on contextInitialized, and would then call the MemacachedClient method shutdown() on contextDestroyed. I would always get a CancellationException or ExecutionException on the first request I would send. (The error messaging alluded to both, but an ExecutionException is what I was able to catch.)

Solution: switched from shutdown() to shutdown(1, TimeUnit.SECONDS)

Now the get call succeeds the very first time that it is run.

I cannot explain for sure how the contextDestroyed call was interfering with the regular handling of the request. My best guess is that spymemcached's single thread somehow gets shared between servlets, and so when a servlet was created to handle a request sent by a verification step of our build process, it would get destroyed prior to the first request I would send, and the MemcachedClient my request's servlet was using would then try to use that same thread and get hit with the exceptions from the shutdown.

(Our team had established the need to call shutdown a while back when we learned our web app had too many open connections to our memcached server.)