Cluster Performance Tuning for Concurrent Requests

Question

I'm trying to optimize my cluster based on the usage profile. In my scenario, I want to ingest historical data only once, then use the cluster only for running queries. On the documentation, it says the default limit for request concurrency is calculated as Cores-Per-Node x 10. Since I want to use the cluster only for querying data, can I increase the limit? Is there a best practice about this?

Answer 1

to my understanding, the documentation answers your questions. specifically - see parts in italic:

Limit on request concurrency

Request concurrency is a limit that a cluster imposes on several requests running at the same time.

• The default value of the limit depends on the SKU the cluster is running on, and is calculated as: Cores-Per-Node x 10.
  • For example, for a cluster that's set up on D14v2 SKU, where each machine has 16 vCores, the default limit is 16 cores x10 = 160.
• The default value can be changed by configuring the request rate limit policy of the default workload group.
  • The actual number of requests that can run concurrently on a cluster depends on various factors. The most dominant factors are cluster SKU, cluster's available resources, and usage patterns. The policy can be configured based on load tests performed on production-like usage patterns.

For more information, see Optimize for high concurrency with Azure Data Explorer.