Bots being stuck in reserved state in storage

Question

This happened in EIL on 12/13/2021: realtime

None of the bots had jobs, confirmed on bot dashboard: dashboard

Answer 1

Apparently a message was lost because there was a K8s upgrade going on and 3 pods were mistakenly called by ops to maintenance. Dispatch had requests still in progress for moving each pod/bot into storage. The solution in this case was the following:

I. Confirm the request still in progress and bot assignation still exist for each bot:

SELECT JobId FROM [iHerb_Scs_Wes_Agv_Dispatch].[request].[Requests]
WHERE Id = (SELECT RequestId FROM[iHerb_Scs_Wes_Agv_Dispatch].[dispatch].[BotAssignments] WHERE BotId = 'c45766cb-21e7-4a91-a509-017cf0e38580')

II. Emit FleetJobCompleted event for each bot, using job id found out on previous step:

https://rabbit-cluster-scs-prod.iherbscs.net/#/queues/scs.wes.agv.dispatch/FleetJobCompleted

{
"JobId" : "EA5E165F-0A72-428B-9B2A-017DB3216120"
}