I have a nightly process that I need to run - once per night. It is a BackgroundService that runs as part of my Blazor app. I can have several instances of the app running (scale out).
Basically I need a semaphore where one of the BackgroundServices grabs it and does the nightly processing. The rest do not. I also need it that if the one that grabbed it then crashes part way through, another will then grab it and complete (the work they do is within transactions hitting the DB so if one dies, another can pick up the work).
So I think I need a semaphore that returns one of:
- Available - caller now has it.
- In process - check back in a bit to see if the worker timed out.
- Work complete - check back tomorrow night.
Is there a way to accomplish this? And if so, how? What I'm really struggling with is handling the server that started on this crashed.