How to make a lot of concurrent web requests using async and await?

Question

I read the how to by Microsoft at How to: Make Multiple Web Requests in Parallel by Using async and await (C#) and found:

private async Task CreateMultipleTasksAsync()  
{  
    // Declare an HttpClient object, and increase the buffer size. The  
    // default buffer size is 65,536.  
    HttpClient client =  
        new HttpClient() { MaxResponseContentBufferSize = 1000000 };  

    // Create and start the tasks. As each task finishes, DisplayResults   
    // displays its length.  
    Task<int> download1 =   
        ProcessURLAsync("http://msdn.microsoft.com", client);  
    Task<int> download2 =   
        ProcessURLAsync("http://msdn.microsoft.com/library/hh156528(VS.110).aspx", client);  
    Task<int> download3 =   
        ProcessURLAsync("http://msdn.microsoft.com/library/67w7t67f.aspx", client);  

    // Await each task.  
    int length1 = await download1;  
    int length2 = await download2;  
    int length3 = await download3;  

    int total = length1 + length2 + length3;  

    // Display the total count for the downloaded websites.  
    resultsTextBox.Text +=  
        string.Format("\r\n\r\nTotal bytes returned:  {0}\r\n", total);  
}  

I understand this code, but my question is: How do I modify this to scale it to like, let's say a hundred or a thousand?

Answer 1

You can invoke async calls in loop. Each call could return a Task, and you have to wait for all Tasks to complete

var requestInfoCollection = new RequestInfo[]
{
     new RequestInfo("http://url1","GET"),
     new RequestInfo("http://url2","GET"),
     new RequestInfo("http://url2","POST")
};
List<Task> tasks = new List<Task>();
foreach(var requestInfo in requestInfoCollection)
{
   tasks.Add(ProcessURLAsync(requestInfo))
}
Task.WaitAll(tasks);

The above will invoke multiple requests and waits for results, however async\await is helpful to release thread to application to use while performing external invocations (http, db etc...). But scaling is depends on your hardware, and your application architecture.

Answer 2

I can't comment due to the restriction on my account, but in reply to @CreativeManix's answer,

List<Task> tasks = new List<Task>();

Task.WaitAll(tasks) will not accept a List of Tasks. One of its override accepts an Array of Tasks.

Task.WaitAll(tasks.ToArray())