I have been load testing an instance of Thingsboard using Jmeter running on one of our Azure VM, I could test the below Scenario without any issue: 5K Threads, each with one HTTP Request (API telemetry) and one data point to one device
But when I started a test for 10K Threads with same HTTP request and data point ,I could see multiple requests throwing error such as
- SSL Handshake Exception: Remote host terminated the handshake
- Broken Pipe (write failed)
- Error 500 etc
After reducing threads per second to 500, I had no issues, so i increased it to 650 and saw a few Requests having errors
I found out about the API Rate limits although there were none set I had this issue, I set the API Rate limit with ample of head room but had no luck , reducing the rate limit to a very small range did affect the error percentage.
Server configurations are: Hardware: Azure D2as (2 vCPUs, 8GB RAM)
Software: Thingsboard docker instance, Single Cassandra DB (no load-balancer used), Kafka Messaging Queue with zookeeper
API call: URL/api/v1/ACCESS_TOKEN/telemetry (also tried with URL/api/plugins/plugins/ENTITY_TYPE/ENTITY_ID/timeseries/ANY?Scope=ANY )
Payload = { "var_1": "NUMBER_BETWEEN_1_AND_1000" }
After trying a few solution I did not see any improvements
Any help on resolving this issue is very much appreciated
Thanks!
PS: I am new to servers and load testing.
Initially, I tried restarting the docker container and server, Had no luck
I tried a local VM instance of same docker image , found no errors
I also tried a different Azure VM already hosted , Had the same issue
Added Rate limits higher than the test cases , did not resolve the issue