I have a set of worker functions that are spun up as needed to pull from Service Bus Topic subscriptions when they are created. New workers are created when a new Subscription is created by way of a provisioning message that is queued triggering a job to spin up the new worker to listen to the subscription. The problem is that now I want to be able to scale out the workers listening to the subscriptions when the app is scaled out. Since the provisioning job only creates the worker on a single instance the effectiveness of scaling out is significantly reduced.
My thought was to create a second provisioning job that runs from a timer trigger to synchronize the running jobs to the current list of subscriptions. I run into the same problem with the timer job as with the service bus trigger though because it is running as a singleton in the web job and likely will run on the same instance of the job each time it is run meaning I still will likely have one maybe 2 instances of a job per subscription no matter how much I scale out.
My question is, is it possible to create a timer job that is not run as a singleton? Meaning, can I configure a timer job that, for each instance of the scaled out web job, will run on a set interval?
Singleton attribute ensures to run only one instance with the help of distributed locking, these are related to webjobs SDK.
Also we have Singleton Listeners and with adding few settings to ensure that your function runs as a singleton on a single instance. To ensure that only a single instance of the function is running when the web app scales out to multiple instances, apply a listener-level singleton lock on the function
([Singleton(Mode = SingletonMode.Listener)]). Listener locks are acquired when the JobHost starts. If three scaled-out instances all start at the same time, only one of the instances acquires the lock and only one listener starts.Now in case of not assigning singleton to avoid running each instance at a time, refer to Multiple Instances documentation from MS Docs