AppFabric Cache errors

Question

The AppFabric Cache in our production crashes almost every day, and is highly unstable. The below errors are logged:

Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (Sufficient secondaries not present or they are in throttled state.)

Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (The request did not find the primary.)

AppFabric Caching service crashed.{Lease with external store expired: Microsoft.Fabric.Federation.ExternalRingStateStoreException: Lease already expired at Microsoft.Fabric.Data.ExternalStoreAuthority.UpdateNode(NodeInfo nodeInfo, TimeSpan timeout) at Microsoft.Fabric.Federation.SiteNode.PerformExternalRingStateStoreOperations(Boolean& canFormRing, Boolean isInsert, Boolean isJoining)}

Could someone please provide me some inputs? This is a HA enabled cache environment with 3 cache hosts. All of them are running on Windows Server 2008 Enterprise Edition, and the SQL Server is used for config.

Answer 1

There are several symptoms of server unavailability. First, applications might receive a DataCacheException exception with an error code of RetryLater and an error substatus of CacheServerUnavailable :

• The cache cluster cannot access the cache configuration store (SQL Server, Xml or Custom)
• The application points to one or more cache hosts that are incorrect, unavailable, or down.
• The cache cluster is down.
• The application runs as a user who has not been granted access to the cache cluster.

Take a look at http://msdn.microsoft.com/en-us/library/ff921010.aspx in order to see the various tools and commands available for monitoring the health of a AppFabric cache cluster

Answer 2

We recently encountered this issue, and it appears to have been due to a defective Read-Through Provider. The Read-Through Provider was not properly managing (closing/disposing) its sql server pooled database connections.

During periods of high usage, we observed the "Lease with external store expired" error as it seems the cache service was actually starved for an available connection to the configuration store.

The Performance Counters for .NET Data Provider for SqlServer were helpful in identifying this issue.