Early EOFException for all clients in Patroni + haproxy cluster

44 Views Asked by At

Cluster consists of: 1 haproxy, 3 postgres. EOFException appears for all client at about 208 of 800 connections (select count(*) used from pg_stat_activity;). What other limits could be causing the problem?

VM Haproxy

ss -tunap | grep 5000 |wc -l
# Output from 170 to 230 (of 4096 limit)
ss -tunap  |wc -l
# Output from 340 to 510 (of 800 limit)
cat /etc/haproxy/haproxy.cfg
global

        maxconn 4096
        log     127.0.0.1 local2

defaults
        log global
        mode tcp
        retries 2
        timeout client 30m
        timeout connect 8s
        timeout server 30m
        timeout check 5s

listen stats
    mode http
    bind *:7000
    stats enable
    stats uri /

listen postgres
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server node1 192.168.19.8:5432 maxconn 200 check port 8008
    server node2 192.168.19.9:5432 maxconn 200 check port 8008
    server node3 192.168.19.10:5432 maxconn 200 check port 8008

Postgres

node-1 (replica)

free -hm
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       388Mi       1.6Gi        59Mi       1.9Gi       3.1Gi
Swap:             0B          0B          0B

node-2 (leader)

free -hm
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       587Mi       1.4Gi       102Mi       1.9Gi       2.9Gi
Swap:             0B          0B          0B
select count(*) used from pg_stat_activity;
# Output to 208, not more
show shared_buffers;
# Output 512MB

node-3

free -hm
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       334Mi       1.4Gi        60Mi       2.1Gi       3.2Gi
Swap:             0B          0B          0B
patronictl -c /etc/patroni.yml edit-config postgres

loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    max_connections: 800
    shared_buffers: 512MB
  pg_hba:
  - host replication replicator 127.0.0.1/32 md5
  - host replication replicator 192.168.19.8/24 md5
  - host replication replicator 192.168.19.9/24 md5
  - host replication replicator 192.168.19.10/24 md5
  - host all all 0.0.0.0/0 md5
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30

EOFException at 800 of 800 connections limit

0

There are 0 best solutions below