We have scylla cluster with tree node. Each node is in its own datacenter. Тo evaluate cluster performance i use cassandra stress test with this parameters:
cassandra-stress write "no-warmup cl=QUORUM" -rate threads=1 -schema "replication(strategy=NetworkTopologyStrategy, DC1=1, DC2=1, DC3=1)" -mode cql3 native connectionsPerHost=80 protocolVersion=4 user=cassandra password=cassandra -node 10.0.0.3
Result:
type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 84, 84, 84, 84, 10.3, 5.0, 45.8, 59.3, 65.9, 65.9, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 145, 61, 61, 61, 16.3, 5.5, 45.1, 46.2, 47.2, 47.2, 2.0, 0.11899, 0, 0, 0, 0, 0, 0
total, 197, 52, 52, 52, 19.3, 5.4, 44.5, 46.0, 46.7, 46.7, 3.0, 0.12423, 0, 0, 0, 0, 0, 0
total, 258, 61, 61, 61, 15.9, 5.3, 44.2, 44.4, 46.0, 46.0, 4.0, 0.09670, 0, 0, 0, 0, 0, 0
total, 325, 67, 67, 67, 15.3, 5.1, 45.4, 46.7, 47.0, 47.0, 5.0, 0.07708, 0, 0, 0, 0, 0, 0
total, 388, 63, 63, 63, 15.9, 5.2, 44.3, 46.2, 47.9, 47.9, 6.0, 0.06482, 0, 0, 0, 0, 0, 0
total, 455, 67, 67, 67, 14.8, 5.3, 46.0, 46.3, 46.5, 46.5, 7.0, 0.05547, 0, 0, 0, 0, 0, 0
total, 518, 63, 63, 63, 15.5, 5.3, 44.0, 46.2, 46.9, 46.9, 8.0, 0.04891, 0, 0, 0, 0, 0, 0
total, 576, 58, 58, 58, 17.2, 5.3, 45.3, 45.8, 48.0, 48.0, 9.0, 0.04541, 0, 0, 0, 0, 0, 0
total, 634, 58, 58, 58, 17.0, 5.3, 44.4, 46.1, 46.2, 46.2, 10.0, 0.04231, 0, 0, 0, 0, 0, 0
total, 704, 70, 70, 70, 14.3, 4.8, 44.1, 46.6, 46.6, 46.6, 11.0, 0.03908, 0, 0, 0, 0, 0, 0
total, 775, 71, 71, 71, 14.4, 4.8, 43.8, 44.1, 46.7, 46.7, 12.0, 0.03650, 0, 0, 0, 0, 0, 0
total, 847, 72, 72, 72, 13.5, 5.0, 41.7, 44.2, 45.7, 45.7, 13.0, 0.03438, 0, 0, 0, 0, 0, 0
total, 914, 67, 67, 67, 15.1, 5.1, 43.8, 43.9, 44.1, 44.1, 14.0, 0.03191, 0, 0, 0, 0, 0, 0
total, 972, 58, 58, 58, 17.0, 5.1, 45.7, 46.6, 47.1, 47.1, 15.0, 0.03089, 0, 0, 0, 0, 0, 0
total, 1031, 59, 59, 59, 17.0, 4.9, 44.2, 44.8, 45.9, 45.9, 16.0, 0.02966, 0, 0, 0, 0, 0, 0
total, 1085, 54, 54, 54, 18.7, 4.8, 44.0, 44.1, 44.7, 44.7, 17.0, 0.02975, 0, 0, 0, 0, 0, 0
total, 1165, 80, 80, 80, 12.1, 4.7, 43.8, 44.3, 45.6, 45.6, 18.0, 0.03075, 0, 0, 0, 0, 0, 0
total, 1224, 59, 59, 59, 17.3, 5.2, 43.9, 46.9, 46.9, 46.9, 19.0, 0.02963, 0, 0, 0, 0, 0, 0
total, 1294, 70, 70, 70, 14.2, 4.8, 43.9, 44.3, 46.4, 46.4, 20.0, 0.02833, 0, 0, 0, 0, 0, 0
total, 1363, 69, 69, 69, 14.2, 4.7, 44.2, 46.7, 46.8, 46.8, 21.0, 0.02707, 0, 0, 0, 0, 0, 0
total, 1432, 69, 69, 69, 14.3, 4.9, 44.0, 44.3, 48.2, 48.2, 22.0, 0.02591, 0, 0, 0, 0, 0, 0
total, 1494, 62, 62, 62, 16.3, 4.9, 44.3, 46.8, 47.3, 47.3, 23.0, 0.02492, 0, 0, 0, 0, 0, 0
total, 1557, 63, 63, 63, 15.7, 5.5, 45.4, 47.3, 47.4, 47.4, 24.0, 0.02395, 0, 0, 0, 0, 0, 0
total, 1620, 63, 63, 63, 16.2, 5.0, 45.2, 46.6, 46.9, 46.9, 25.0, 0.02305, 0, 0, 0, 0, 0, 0
total, 1687, 67, 67, 67, 14.9, 4.8, 44.1, 45.5, 45.6, 45.6, 26.0, 0.02216, 0, 0, 0, 0, 0, 0
total, 1743, 56, 56, 56, 17.8, 4.8, 44.0, 46.3, 47.3, 47.3, 27.0, 0.02203, 0, 0, 0, 0, 0, 0
total, 1804, 61, 61, 61, 16.0, 4.8, 43.8, 44.4, 45.4, 45.4, 28.0, 0.02138, 0, 0, 0, 0, 0, 0
total, 1877, 73, 73, 73, 13.5, 4.8, 44.1, 44.5, 44.8, 44.8, 29.0, 0.02101, 0, 0, 0, 0, 0, 0
total, 1941, 64, 64, 64, 15.9, 5.0, 44.0, 46.7, 47.3, 47.3, 30.0, 0.02032, 0, 0, 0, 0, 0, 0
total, 2003, 62, 62, 62, 16.0, 4.8, 44.0, 44.1, 44.2, 44.2, 31.0, 0.01975, 0, 0, 0, 0, 0, 0
total, 2004, 215, 215, 215, 16.9, 16.9, 16.9, 16.9, 16.9, 16.9, 31.0, 0.07564, 0, 0, 0, 0, 0, 0
As we see, we have a relatively small mead and mean latency, but catastrophic .95 and .99 latency. And main question: why? And why this values so different?
It's not a goo practice to have a single node per data center. It could be the source of latency. You also run a single thread in the client but that's more of a throughput issue. There are no details about the machines you run. Try to replicate one of the published benchmarks of Scylla