How can I optimize the performance of my pulsar cluster

Question

Hardware

I have two hard drives, one is SSD and one is HD, both with a capacity of more than 100G, and the CPU is 8 cores and 64G.The read/write speed of the disk is about 100 MB/s.I deployed broker and bookie separately on 6 servers.Pulsar version is 3.1.1

Configuration

I use SSD for Ledgers and HD for Journal,other configs use the default configuration of pulsar. I use the OpenMessaging Benchmark framework for testing, and Prometheus and Grafana monitor the system

Problems

1. Processing of persistent messages is very slow My Pulsar cluster performs well when processing non-persistent messages, with a pub latency of 0.7~5ms, but the max latency may reach 5000ms when processing persistent messages. enter image description here

2. The consumer is very slow to respond When using the OMB framework for testing, the response of the consumer is very slow

3. Performance decreases significantly after message increases When persistent storage is used, the performance of a 24Kb message is significantly lower than that of 1Kb.Besides,I don't think Pulsar should have this level of performance with my hardware configuration enter image description here

4. Other tools for monitoring system performance I'm now using the iostat command to check the performance of the disk, the top command to check the CPU, and hopefully there are other checking tools

5. The reason why persisting messages in pub laterncy is so great and how to optimize it