How can I avoid duplicate background task processing in Service Fabric hosted services?

Question

Sorry about the vague title, it's rather hard to explain. I have the following setup:

• I'm running a .NET Core 2.2 Web API hosted in Service Fabric.
• Part of this API's responsibilities is to monitor an external FTP storage for new incoming files.
• Each file will trigger a Mediator Command to be invoked with processing logic.
• I've implemented a hybrid solution based on https://learn.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/background-tasks-with-ihostedservice and https://blog.maartenballiauw.be/post/2017/08/01/building-a-scheduled-cache-updater-in-aspnet-core-2.html. In essence this is an IHostedService implementation that is registered in the Startup.cs of this API. Its basically a background service running in-process.

As for the problem. The solution above works fine on a 1-node cluster, but causes "duplicates" to be processed when running on a 5-node cluster. The problem lies in the fact that on a 5-node cluster, there are ofcourse 5 identical ScheduledTasks running and will all access the same file on the FTP at the same time.

I've realised this is caused somewhat by improper separation of concerns - aka the API shouldn't be responsible for this, rather a completely separate process should handle this.

This brings me to the different services supported on Service fabric (Stateful, Stateless, Actors and Hosted Guest Exe's). The Actor seems to be the only one that runs single-threaded, even on a 5-node cluster. Additionally, an Actor doesn't seem to be well suited for this kind of scenario, as it needs to be triggered. In my case, I basically need a daemon that runs all the time on a schedule. If I'm not mistaken, the other stateful/stateless services will run with 5 "clones" as well and just cause the same issue as I currently have.

I guess my question is: how can I do efficient background processing with Service Fabric and avoid these multi-threaded/duplicate issues? Thanks in advance for any input.

Answer 1

In service farbic you have 2 options with actors:

• Reliable actor timers
• Reliable actor reminders

You can use the state to determine if the actor has processed your ftp file.

Have a look at this blog post, to see how they used a reminder to run every 30 seconds.

It's important that the code in your actor allows reantrancy. Basically because the actors are reliable, your code might get executed multiple times and be canceled in the middle of an execution.

Instead of doing this:

public void Method()
{
    _ftpService.Process(file);
}

Consider doing this:

public void Method(int fileId)
{
    if (_ftpService.IsNotProcessed(fileId))
    {
        _ftpService.Process(file);
        _ftpService.SetProcessed(fileId);
    }
}

If your actor has trouble disposing, you might want to check if you are handling cancelationtokens in your code. I never had this issue, but we are using autofac, with Autofac.ServiceFabric to register our actors with RegisterActor<T>() and we have cancelationtokens in most of our logic. Also the documentation of CancellationTokenSource can help you. Example

public Ctor()
{
    _cancelationTokenSource = new CancellationTokenSource();
    _cancellationToken= _cancelationTokenSource.Token;
}

public async Task SomeMethod()
{
    while(/*condition*/)
    {
       _cancellationToken.ThrowIfCancellationRequested();
       /*Other code*/
    }
}

protected override async Task OnDeactivateAsync()
{
    _cancelationTokenSource.Cancel();
}