Apache httpd (mod_proxy) seems to drop/ignore 3rd IP address in X-Forwarded-For chain?

Question

Consider the scenario/flow:

remote user (client) > proxy1 > proxy2 > AWS ALB > httpd/reverse_proxy > my_application

As user's request traverses out of their network, the X-Forwarded-For (XFF) header is appended with IP address of each successive proxy. For example, when it arrives to the ALB the XFF header contains "192.168.1.100, 100.99.98.97". ALB will then append the ClientIP to this header, and in this case that is the IP of proxy2. Finally, when request arrives to the reverse proxy sitting in front of my app, the XFF header is now: "192.168.1.100, 100.99.98.97, 95.94.93.92".

Issue I'm seeing: At the reverse_proxy , httpd seems to be ignoring or dropping the last/most right IP in the X-Forwarded-For header chain, specifically when there are more than 2 addresses.

• tcpdump at reverse proxy shows the header contains full chain, so AWS ALB is doing what it should and appending proxy2's address.
• when printing the XFF header in httpd access log, I only see the first 2 addresses in the chain printed.
• more importantly, when trying to take action on the expected third address, this tests fails, further proving (I think), the 3rd address is dropped. By 'take action' I mean trying to invoke Require directive (mod_authz_host), after setting RemoteIPHeader X-Forwarded-For in the vhost.
• If I remove proxy1 from the flow, then I see address for proxy2 in the XFF header at the reverse proxy without issue.

I'm not sure what I missed in my config or testing, and while not a standard, multiple addresses in XFF header is common. I face this issue only in Apache httpd 2.4. In version 2.2, this wasn't an issue and I can repeat the same setup/flow and I see all 3 addresses in the chain. Thanks in advance.

example vhost:

<VirtualHost *:80>
  ProxyPreserveHost On
  ServerName myapp.mydomain.com
  Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains;"
  RemoteIPHeader X-Forwarded-For


  ProxyPass / http://10.1.2.3:8080/  timeout=3600
  ProxyPassReverse "/" http://10.1.2.3:8080/

  SetEnv proxy-sendchunked

  ErrorLog /var/log/httpd/error_myapp
  LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined2
  CustomLog /var/log/httpd/access_myapp combined2
  </VirtualHost>

Answer 1

Vhost is missing RemoteIPTrustedProxy directive.

"When multiple, comma delimited useragent IP addresses are listed in the header value, they are processed in Right-to-Left order. Processing halts when a given useragent IP address is not trusted to present the preceding IP address. The header field is updated to this remaining list of unconfirmed IP addresses, or if all IP addresses were trusted, this header is removed from the request altogether."

https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy

The answer was right in front of me, as usual.