I'm trying to benchmark our Aerospike cluster of 3 nodes with build-in Java benchmark tool.
When I check AMC (Aerospike Management Console) I see there are no successful reads. Benchmark is reporting that everything is ok and that's the problem.
2016-12-16 15:34:21.674 write(tps=14851 timeouts=0 errors=0) read(tps=15030 timeouts=0 errors=0) total(tps=29881 timeouts=0 errors=0)
2016-12-16 15:34:22.674 write(tps=21160 timeouts=0 errors=0) read(tps=21284 timeouts=0 errors=0) total(tps=42444 timeouts=0 errors=0)
2016-12-16 15:34:23.675 write(tps=22868 timeouts=0 errors=0) read(tps=22312 timeouts=0 errors=0) total(tps=45180 timeouts=0 errors=0)
2016-12-16 15:34:24.676 write(tps=22443 timeouts=0 errors=0) read(tps=22795 timeouts=0 errors=0) total(tps=45238 timeouts=0 errors=0)
Do you have any idea why AMC do not show those read requests as successful since benchmark reports reading with no error or timeout?
My benchmark configuration (modified example 3) is bellow.
./run_benchmarks -h aero1.db.test.env -p 3000 -n namespace -k 1000000000000000 -S 1 -o S:50 -w RU,50 -z 1 -async -asyncMaxCommands 300 -asyncSelectorThreads 8 -e 1 -T 500
That's a lot of keys you have there (1 Million Billion keys, or 10^15)... and doing an RU workload...
Your read failures are in fact 'not found'. I am not a statistician, but I would expect the chances of randomly reading a key that has been already inserted from such a large sample to be very very low for a very very long time. You can check the stats and verify that all those reads are indeed 'not found'. Those are technically reported separately from the error in the benchmark tool, which is, I give you that, misleading at best since you wouldn't notice.
I am also not sure what type of RAM/Storage you have on a 3 node cluster to handle that many records, as small as they are.
For proper benchmark testing, you should first load all your keys with an insert only workload (-w I) and then trigger the read update workload.