I have an Azure function set up to receive MS Graph webhook notifications, and a few times a day it gets relatively heavy load (1K req/sec for a few seconds). I'm using a distributed Redis lock to limit the actual handling of those events, but often receive timeouts. I've read the recommended guidance and I'm not seeing an obvious issue from the error message. I've made sure to use a singleton connection and I increased the thread pool count to 200. What am I missing?
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5156ms elapsed, timeout is 5000ms), command=HMGET, next: HMGET CacheKeyRedacted, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 5316, last-in: 1, cur-in: 0, sync-ops: 0, async-ops: 10796, serverEndpoint: redacted.redis.cache.windows.net:6380, conn-sec: 132342.43, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: redacted(SE.Redis-v2.7.4.20928), IOCP: (Busy=1,Free=999,Min=200,Max=1000), WORKER: (Busy=3,Free=1020,Min=200,Max=1023), POOL: (Threads=27,QueuedItems=4,CompletedItems=2499702,Timers=134), v: 2.7.4.20928 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Try adding the following settings to your connection string: