SQL Server service brokers waitfor c# httpclient long open http

Question

I am working on a system that uses HTTPClient to get a HTTP route like /api/GUID and holds the connection open until another system sends a message to SQL Server to notify that the work is completed. Once the message is received the connection ends and my client continues on with its data it was requesting.

When I test this with curl or postman I get the intended results.

I get intermittent issues with c# httpclient. Sometimes I get my data immediately. Other times httpclient will stay open and after its timeout expires the message is shown in my test app despite throwing a task cancellation exception. Other times nothing is returned and a timeout occurs OR I only receive an empty string from my data. This very well could be related to our service broker implementation but i want to rule out everything about HTTPclient. I have tried using threads/tasks/ lots of async await using.Result and I am getting the same results so far. It is highly likely related to our servicebroker waitfor but any insight would be great. The issue tends to be connections that come in rapidly from a single source tend to have this pattern.

public string GetSynchronous(string url)
{
        var tokenSource = new CancellationTokenSource();
        var cancellationToken = tokenSource.Token;
        return _httpClient.GetAsync(url, cancellationToken).Result.Content.ReadAsStringAsync().Result;
}

public void PostAsync(string url, JObject jsonBody)
{
        var content = new StringContent(jsonBody.ToString(), Encoding.UTF8, "application/json");
        var tokenSource = new CancellationTokenSource();
        var cancellationToken = tokenSource.Token;
        var result = _httpClient.PostAsync(url, content, cancellationToken).Result;
}

private static readonly HttpClient _httpClient = new HttpClient {
        Timeout = TimeSpan.FromSeconds(60),
        DefaultRequestHeaders =
{
        Accept = { new MediaTypeWithQualityHeaderValue("application/json")},
        ConnectionClose = true
    },
};


class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hit enter to start");
        Console.ReadLine();
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
        ServicePointManager.DefaultConnectionLimit = 100;
        var tests = new Tests();
        tests.CallSynchronizer();
        tests.CallSynchronizerPostFirst();
        Console.WriteLine("Hit enter to end");
        Console.ReadLine();
    }
}

Answer 1

Service Broker is designed for long exchanges (hours, days, weeks). Waiting for a response in the HTTP requests is definitely an anti-pattern. And more fundamentally, if concurrent requests are using this patter, there is nothing preventing them from each receiving the response intended for the other request.

You HTTP requests should post the work to-do (ie. issue a SEND) and return immediately. The response, when it comes, should activate a stored procedure and this should update some state, write the result. From HTTP you could then poll for the result and return current status immediately. Your app should work with service broker responses that return after a week same way as if the response returned in 10ms.

Answer 2

Ultimately this ended up being related to how we are load balancing our application and sqldependency.start queue name being set. This is a service that is setup in an anti-pattern way to migrate old code off of a really unreliable set of services.