How can I control thread count when I use "Task.WhenAll"

Question

I am verifying image urls by making an http get request asynchronously. All works fine with the code below but when I have so many Images, our firewall will block my internet access because of so many threads concurrently requesting. Therefore I was looking for a solution how to restrict the count of concurrently running threads. I ended up with this thread telling me to use SemaphoreSlim but I am somehow not able to get the idea and how to implement this?

• is that SemaphoreSlim wait or waitAsnyc (what is the difference anyway?) should be inside a foreach while adding tasks? Can I just create the task list with linq as I do in my code?
  • why is there used task.Run?
  • after which line is executed does the thread start? after task.run or task.whenall?

If that's not the best approach, please suggest a better one. I'm not sure if using MaxDegreeOfParallelism with parallel.foreach makes sense as well?

  Dim tasks = myImages.Select(Function(x) testUrl_async(x))
  Dim results = Await Task.WhenAll(tasks)

Async Function testUrl_async(ByVal myImage  As image) As Task(Of image)
   Dim myImageurl as string=myImage.imageurl
   myHttpResponse = Await myHttpClient.GetAsync(myImageurl)
    If myHttpResponse.IsSuccessStatusCode Then
        Return myImage
    Else
        Return Nothing
    End If
End Function

Answer 1

our firewall will block my internet access because of so many threads concurrently requesting. Therefore I was looking for a solution how to restrict the count of concurrently running threads.

Pretty sure that the firewall is restricting your number of connections, and thus you want to restrict your number of connections (not threads).

is that SemaphoreSlim wait or waitAsnyc (what is the difference anyway?)

Wait is a synchronous wait - it blocks the calling thread. WaitAsync is an asynchronous wait - it frees the calling thread and resumes executing the current method when the semaphore is available.

should be inside a foreach while adding tasks? Can I just create the task list with linq as I do in my code?

You can do it either way: build up a list explicitly, or use LINQ.

why is there used task.Run?

That's an error in that answer. Task.Run is certainly not needed or desired here.

after which line is executed does the thread start? after task.run or task.whenall?

When you call Task.Run, that delegate is queued to the thread pool immediately. But as I said above, you don't want to use Task.Run (it also shouldn't be used in the original answer either).

---

So, something like this should suffice:

Private _mutex As New SemaphoreSlim(20)
Async Function testUrl_async(myImage As image) As Task(Of image)
    Await _mutex.WaitAsync()
    Try
        Dim myImageurl = myImage.imageurl
        Dim myHttpResponse = Await myHttpClient.GetAsync(myImageurl)
        Return If(myHttpResponse.IsSuccessStatusCode, myImage, Nothing)
    Finally
        _mutex.Release()
    End Try
End Function

Answer 2

You can use Task and SemaphoreSlim like this:

var MaxAllowedConcurrentRequestsCount = 20;
var guard = new SemaphoreSlim(0, MaxAllowedConcurrentRequestsCount);
foreach(var imageUrl in imageUrls)
{
    guard.Wait()
    var image = await Task.Factory.StartNew((imageUrl) => return myHttpClient.Get(imageUrl));
    guard.Release();
    // You can do whaterver you like with image. For example add it to you concurrent list.
}

Answer 3

I assume that you are not inside the WPF or the Windows Form Thread. If you are there will be only one thread working after the await.

As we assume you are not in those threads, the ThreadPool is used to execute the continuations after the await. You can change the amount of Threads used by the pool with ThreadPool.SetMaxThreads but I suggest you don't do that and leave it up to .NET to do the right thing. That is usually the best way.