Hangfire running recurring jobs on different web servers

Question

We have multiple web servers running our sites, with load balancers, so users are directed to different servers depending on the load. The code is the same for each instance on each server, and we have recurring jobs that are run. Obviously we dont want the jobs to run at the same time on both servers.

Does hangfire implement a lock when a job is run so it is not run again automatically?

Currently we have this already on each method that is run [Hangfire.DisableConcurrentExecution(60 * 60 * 5)] will that stop both servers running the code at the same time?

Answer 1

In order to get around this in the end, i did the following

These attributes are added to my interfaces

[PingUrlToKeepAlive]
[SkipWhenPreviousJobIsRunning]
[DisableConcurrentExecution(10)]
[Queue("{0}")]

PingUrl is an attribute created to stop the IIS process from shutting down after 20 minutes on either server, nothing to do with this fix, just thought i would mention it

Queue is the recommended way now according to hangfire.

DisableConcurrentExecution is the attribute i thought i needed only, but you also need the one below.

SkipWhenPreviousJobIsRunning is a new attribute, that looks like this

public class SkipWhenPreviousJobIsRunningAttribute: JobFilterAttribute, IClientFilter, IApplyStateFilter
{
    public void OnCreating(CreatingContext context)
    {
        var connection = context.Connection as JobStorageConnection;

        // We can't handle old storages
        if (connection == null) return;

        // We should run this filter only for background jobs based on 
        // recurring ones
        if (!context.Parameters.ContainsKey("RecurringJobId")) return;

        var recurringJobId = context.Parameters["RecurringJobId"] as string;

        // RecurringJobId is malformed. This should not happen, but anyway.
        if (string.IsNullOrWhiteSpace(recurringJobId)) return;

        var running = connection.GetValueFromHash($"recurring-job:{recurringJobId}", "Running");
        if ("yes".Equals(running, StringComparison.OrdinalIgnoreCase))
        {
            context.Canceled = true;
        }
    }

    public void OnCreated(CreatedContext filterContext)
    {
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        if (context.NewState is EnqueuedState)
        {
            var recurringJobId = SerializationHelper.Deserialize<string>(context.Connection.GetJobParameter(context.BackgroundJob.Id, "RecurringJobId"));
            if (string.IsNullOrWhiteSpace(recurringJobId)) return;

            transaction.SetRangeInHash(
                $"recurring-job:{recurringJobId}",
                new[] { new KeyValuePair<string, string>("Running", "yes") });
        }
        else if (context.NewState.IsFinal /* || context.NewState is FailedState*/)
        {
            var recurringJobId = SerializationHelper.Deserialize<string>(context.Connection.GetJobParameter(context.BackgroundJob.Id, "RecurringJobId"));
            if (string.IsNullOrWhiteSpace(recurringJobId)) return;

            transaction.SetRangeInHash(
                $"recurring-job:{recurringJobId}",
                new[] { new KeyValuePair<string, string>("Running", "no") });
        }
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
    }
}

Basically this checks to see if the job is already running and if so, cancels it. We now have no problems with jobs running on both servers at the same time.

Answer 2

Take a look on Mutexes. They will prevent the execution two processes in the same time which need access to one resource.

Mutex prevents concurrent execution of multiple background jobs that share the same resource identifier. Unlike other primitives, they are created dynamically so we don’t need to use IThrottlingManager to create them first. All we need is to decorate our background job methods with the MutexAttribute filter and define what resource identifier should be used.

[Mutex("my-resource")]
public void MyMethod()
{
    // ...
}

When we create multiple background jobs based on this method, they will be executed one after another on a best-effort basis with the limitations described below. If there’s a background job protected by a mutex currently executing, other executions will be throttled (rescheduled by default a minute later), allowing a worker to process other jobs without waiting.

Source: https://docs.hangfire.io/en/latest/background-processing/throttling.html#