Azure Redis timeouts

Question

I have a Asp.Net Azure web app calling a Azure Redis instance. I keep getting timeouts on Redis. The message I get is below.

 inst: 1, mgr: Inactive, err: never, queue: 120, qu: 0, qs: 120, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, clientName: RD0004FFA37AA4, serverEndpoint: Unspecified/server.redis.cache.windows.net:6380, keyHashSlot: 11524, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=21,Free=8170,Min=200,Max=8191)

Both the app and redis are in the same region (East US 2).

Answer 1

Having "in: 65536" in your error message indicates that there are 65536 bytes sitting in the client kernel socket receive buffer ready to be processed but have not yet been parsed by the client app.

I have seen this in two cases:

1. ThreadPool settings need to be adjusted (as described here). However, the error message provided above does not show this as being a problem. Check a sampling of more errors to ensure that this is not happening in other cases.
2. Client side CPU is running hot. When the CPU is high, the code that handles the socket receive events is not triggered in a timely manner, so the processing code does not parse the waiting data in the kernel buffer. Check your client CPU history. Be careful though - a short-lived spike in CPU might not show up in azure metrics because the CPU is captured on a periodic cycle (I think every 20 seconds). If the spike happens between those samples and is short, it might not be noticed by the metrics gathering system.

Other common client-side causes are documented here and common server-side errors are documented here.