When copying data from REST API with different endpoints.
This one is very slow getting throughput of 20kb/s www.mydata/products
but these ones are running fine getting 5mb/s throughput www.mydata/customer www.mydata/salesTransactions
I need to find a way to drastically increase throughput of the products endpoint of my API Connection.
The copy throughput remains at below 20KB/s,
I tried almost everything from increasing the block size.
I Tried increasing the amount of records per page for the API parameter.
I Increased the DUI's and parallelism
it still uses 4 DUI's and 1 degree of parallelism.
I have a custom Azure Runtime with a corecount of 16 (I tried 32 corecount as well)
.
I tried everything out of the box with the copy activity but nothing increases the throughput.
My last resort is to have 30 or 31 copy activities in one pipelines with each one running from and to dates as each day of the month running them concurrently and see if I get better results. Are there any good articles on copying data from REST Api's where I could use iterations activities to test this?