Nginx server request timeout for Safari with iCloud Private Relay

Question

I have a nginx ssl reverse proxy server and for some reason it doesn't work with Safari with iCloud Private Relay and works fine with other browsers or if I disabled iCloud Relay.

Here is a reduced config

error_log /nginx.log debug;

server {
        listen 80;
        listen [::]:80;

        location / {
                rewrite ^ https://$host$request_uri? permanent;
        }

        location ~ /.well-known/acme-challenge {
                allow all;
                root /var/www/html;
        }
}

server {
        listen 443 ssl;

        ssl_certificate /etc/letsencrypt/live/xxx.xxx.xxx/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/xxx.xxx.xxx/privkey.pem;

        location / {
               return 404;
        }
}

and I found this in the log

2023/12/04 23:08:43 [info] 28#28: *77 client timed out (110: Connection timed out) while waiting for request, client: xxx.xxx.xxx.xxx, server: 0.0.0.0:443

Non ssl 80 port accessing is completely fine.

Is there any good way to debug this issue? Obviously I am still able to browse the internet with Safari so it is a bit unlikely it is the iCloud Private Relay is broken. But maybe some ssl cipher combination that is troublesome?