Task.WaitAll and AggregateException handling

59 Views Asked by At

I've got a little program that needs to update a lot of row statuses with async batched api calls.

I've got an HttpClient instance created with all the constants of the api call set, and then I launch the calls asynchronously in batches of 100. Then I do a Task.WaitAll() for the whole set.

I put a catch(AggregateException) block around Task.WaitAll() to log if there were any errors, then I try to process the ones that succeeded.

I've started getting errors lately (usually timeouts/cancels), but the thing that has me scratching my head is that the bottom "process the successes" loop is throwing another AggregateException.

The Task.WaitAll() throws one with, say, 2 or 3 timeouts out of 100 requests. Then the bottom loop that's supposed to process the successes throws another one with a single InnerException.

My guess is that either

  1. When there's an AggregateException on Task.WaitAll(), it might not be actually waiting for all 100 to be complete or
  2. The second loop touching apiResult.Exception != null (or the 2nd if with apiResult.Exception == null && apiResult.Result != null) is causing another AggregateException to be thrown with the single task being examined.

Is looking at asiResult.Exception the wrong way to see if a task errored out?

Thanks

    private static async Task<Tuple<ApiStatus, long>> LaunchApiRequest(string id)
    {
        var result = ApiStatus.Unknown;
        var sw = Stopwatch.StartNew();
        // This approach throws exception for any non-200 response, which is expensive.  Unlike WebRequest, the exception doesn't give you a respose object.  
        // This api returns json describing the error so we need to make the call and the deserialization separate steps to the errors in line with the code.
        // var json = await _httpClient.GetFromJsonAsync<JsonElement>("?newsID=" + id); 
        var req1 = new HttpRequestMessage(HttpMethod.Get, "?ID=" + id);
        var resp = await _httpClient.SendAsync(req1);
        sw.Stop();
        var reqTime = sw.ElapsedTicks;
        if (resp.StatusCode == HttpStatusCode.NotFound)
            result = ApiStatus.NotFound;
        else if (!resp.IsSuccessStatusCode)
            result = ApiStatus.Error;
        else
        {
            var json = await JsonSerializer.DeserializeAsync<JsonElement>(await resp.Content.ReadAsStreamAsync());    // System.Text.Json
            // look at json response here
        }

        return new Tuple<ApiStatus, long>(result, reqTime);
    }

...

    try
    {
         // loop construct to get more batches
...
                    int asyncExceptions = 0;
                    try
                    {
                        Task.WaitAll(batchRequests);
                    }
                    catch(AggregateException e)
                    {
                        asyncExceptions = e.InnerExceptions.Count;
                        var firstE = e.InnerExceptions[0];
                        _logger.Error($"{asyncExceptions.ToString()} api requests out of {batchRequests.Length.ToString()} threw exceptions.  Sample exception: {firstE.Message}\nStackTrace: {firstE.StackTrace}");
                    }

                    if (asyncExceptions == batchRequests.Length)
                    {   // All requests in this batch errored out; api must be down
                        throw new Exception("All api requests in batch returned errors.  Must be a problem with the api.");
                    }

                    long apiBatchTimeTotal = 0, apiBatchTimeMin = long.MaxValue, apiBatchTimeMax = long.MinValue;
                    int countErrs = 0, count404s = 0;
                    r = 0;
                    int lastGoodRequest = 0;
                    foreach (var apiResult in batchRequests)
                    {
                        if (apiResult.Exception != null || apiResult.Result == null || apiResult.Result.Item1 ==  ApiStatus.Error) countErrs++;
                        else if (apiResult.Result.Item1 == ApiStatus.NotFound) count404s++;

                        if (countErrs == 0 && count404s == 0) lastGoodRequest = r;  // if we haven't hit any 404s or errors this batch, move the stake in the sand up to here.

                        if (apiResult.Exception == null && apiResult.Result != null)
                        {
                            var callTime = apiResult.Result.Item2;
                            if (callTime < apiBatchTimeMin) apiBatchTimeMin = callTime;
                            if (callTime > apiBatchTimeMax) apiBatchTimeMax = callTime;
                            apiBatchTimeTotal += callTime;
                        }
                        r++;
                    }
...
        }
        catch (AggregateException e)
        {
            var firstE = e.InnerExceptions[0];
            _logger.Fatal($"{e.InnerExceptions.Count.ToString()} api requests threw exceptions.  Sample exception: {firstE.Message}\nStackTrace: {firstE.StackTrace}");
            SendEMailMessage(EmailRecipients, "Batch Api Status Failure!", $"Sample Error: {firstE.Message}\nStackTrace: {firstE.StackTrace}", "High");
        }
0

There are 0 best solutions below