I've got a little program that needs to update a lot of row statuses with async batched api calls.
I've got an HttpClient instance created with all the constants of the api call set, and then I launch the calls asynchronously in batches of 100. Then I do a Task.WaitAll() for the whole set.
I put a catch(AggregateException) block around Task.WaitAll() to log if there were any errors, then I try to process the ones that succeeded.
I've started getting errors lately (usually timeouts/cancels), but the thing that has me scratching my head is that the bottom "process the successes" loop is throwing another AggregateException.
The Task.WaitAll() throws one with, say, 2 or 3 timeouts out of 100 requests. Then the bottom loop that's supposed to process the successes throws another one with a single InnerException.
My guess is that either
- When there's an AggregateException on Task.WaitAll(), it might not be actually waiting for all 100 to be complete or
- The second loop touching apiResult.Exception != null (or the 2nd if with apiResult.Exception == null && apiResult.Result != null) is causing another AggregateException to be thrown with the single task being examined.
Is looking at asiResult.Exception the wrong way to see if a task errored out?
Thanks
private static async Task<Tuple<ApiStatus, long>> LaunchApiRequest(string id)
{
var result = ApiStatus.Unknown;
var sw = Stopwatch.StartNew();
// This approach throws exception for any non-200 response, which is expensive. Unlike WebRequest, the exception doesn't give you a respose object.
// This api returns json describing the error so we need to make the call and the deserialization separate steps to the errors in line with the code.
// var json = await _httpClient.GetFromJsonAsync<JsonElement>("?newsID=" + id);
var req1 = new HttpRequestMessage(HttpMethod.Get, "?ID=" + id);
var resp = await _httpClient.SendAsync(req1);
sw.Stop();
var reqTime = sw.ElapsedTicks;
if (resp.StatusCode == HttpStatusCode.NotFound)
result = ApiStatus.NotFound;
else if (!resp.IsSuccessStatusCode)
result = ApiStatus.Error;
else
{
var json = await JsonSerializer.DeserializeAsync<JsonElement>(await resp.Content.ReadAsStreamAsync()); // System.Text.Json
// look at json response here
}
return new Tuple<ApiStatus, long>(result, reqTime);
}
...
try
{
// loop construct to get more batches
...
int asyncExceptions = 0;
try
{
Task.WaitAll(batchRequests);
}
catch(AggregateException e)
{
asyncExceptions = e.InnerExceptions.Count;
var firstE = e.InnerExceptions[0];
_logger.Error($"{asyncExceptions.ToString()} api requests out of {batchRequests.Length.ToString()} threw exceptions. Sample exception: {firstE.Message}\nStackTrace: {firstE.StackTrace}");
}
if (asyncExceptions == batchRequests.Length)
{ // All requests in this batch errored out; api must be down
throw new Exception("All api requests in batch returned errors. Must be a problem with the api.");
}
long apiBatchTimeTotal = 0, apiBatchTimeMin = long.MaxValue, apiBatchTimeMax = long.MinValue;
int countErrs = 0, count404s = 0;
r = 0;
int lastGoodRequest = 0;
foreach (var apiResult in batchRequests)
{
if (apiResult.Exception != null || apiResult.Result == null || apiResult.Result.Item1 == ApiStatus.Error) countErrs++;
else if (apiResult.Result.Item1 == ApiStatus.NotFound) count404s++;
if (countErrs == 0 && count404s == 0) lastGoodRequest = r; // if we haven't hit any 404s or errors this batch, move the stake in the sand up to here.
if (apiResult.Exception == null && apiResult.Result != null)
{
var callTime = apiResult.Result.Item2;
if (callTime < apiBatchTimeMin) apiBatchTimeMin = callTime;
if (callTime > apiBatchTimeMax) apiBatchTimeMax = callTime;
apiBatchTimeTotal += callTime;
}
r++;
}
...
}
catch (AggregateException e)
{
var firstE = e.InnerExceptions[0];
_logger.Fatal($"{e.InnerExceptions.Count.ToString()} api requests threw exceptions. Sample exception: {firstE.Message}\nStackTrace: {firstE.StackTrace}");
SendEMailMessage(EmailRecipients, "Batch Api Status Failure!", $"Sample Error: {firstE.Message}\nStackTrace: {firstE.StackTrace}", "High");
}