TPL BufferBlock.ReceiveAsync receiving same item twice

Question

I have a BufferBlock setup like this.

_inputQueue = new BufferBlock<WorkItem>(new DataflowBlockOptions
{
    BoundedCapacity = 1,
    CancellationToken = cancellationToken,
    EnsureOrdered = true
});

Have multiple consumers calling the "FetchWork" function from separate threads

public async Task<WorkItem> GetWork()
{
    WorkItem wi;
    try
    {
        wi = await _inputQueue.ReceiveAsync(new TimeSpan(0, 0, 1));
    }
    catch (Exception)
    {
        //since we supplied a timeout, this will be thrown if no items come back
        return null;
    }
    return wi;
}

Occasionally, the same workitem ends up in multiple consumers! The more the number of workitems in InputQueue", the more the chances of duplicates received in GetWork. My understanding is items fetched via ReceiveAsync are atomic and once an item is read, will not be read again. That's not happening here. I have around 40 parallel consumers calling GetWork.

Answer 1

This seems to be a service fabric issue. The BufferBlock is dequeing the item only once. The producers [service fabric stateful service instances with partition count of 5] are receiving the same item twice in different partitions. Have to investigate this.