Assume a task that has to be executed by a number of hardware devices. The task (detecting an electrical failure that might cost human lives and is thus) is very important and must reliably be executed regularly. Every time one of the devices executes the task, the others must not do so. The devices are controlled by small computers. There might just be a single device/controller pair, or any number of them which need to coordinate. Controllers communicate by sending messages via a broker. Sending messages takes varying time, messages might get lost, device/controller pairs might come in and drop out unexpectedly at any time and with no warning. The controllers are rather weak embedded machines. Yet each of them must execute the task at least once every ${TIME}, and none of the others must do so at the same time.
What would be a good, simple, and reliable protocol for the brokers to sync with each other so that the above criteria are fulfilled?