I am constructing my reqwest::Client using a builder and set a proxy. My objective it to load-test the proxy.
The load tests spawn multiple futures that all use the same (cloned) Client. (I don't think this is relevant for the problem described below, but I mention it for completion.)
My assumption was, that when I specify a proxy on the reqwest client, each request would be intercepted by the proxy. This is true as long as the client starts a new connection. However, if the client falls back to a pooled connection then the request is not being intercepted by the proxy, but appears to made against the final destination directly.
When sending thousands of requests only about a third actually hit the proxy - which defeats the purpose of my load test.
I tried to come up with a connection pool policy that would enforce the client to start a new connection. But, besides not being able to properly achieve that, this is not what I want.
Here is part of the logs that show the behavior where only some requests to different endpoints at https://www.test.local:8081 get intercepted by the proxy at 10.12.128.5:3128:
2024-01-18T20:48:28.016897686+11:00 DEBUG hyper::client::pool - pooling idle connection for ("https", www.test.local:8081)
2024-01-18T20:48:28.016948690+11:00 DEBUG e2e_tests::utils - Request to https://www.test.local:8081/endpoint-7 finished after: 388 ms
2024-01-18T20:48:28.016991381+11:00 DEBUG e2e_tests::utils - Response: Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.test.local")), port: Some(8081), path: "/endpoint-7", query: None, fragment: None }, status: 200, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Thu, 18 Jan 2024 09:48:27 GMT"} }
2024-01-18T20:48:28.017498236+11:00 DEBUG rustls::common_state - Sending warning alert CloseNotify
2024-01-18T20:48:28.020785046+11:00 DEBUG hyper::proto::h1::io - parsed 3 headers
2024-01-18T20:48:28.020828067+11:00 DEBUG hyper::proto::h1::conn - incoming body is content-length (10 bytes)
2024-01-18T20:48:28.020865719+11:00 DEBUG hyper::proto::h1::conn - incoming body completed
2024-01-18T20:48:28.020980953+11:00 DEBUG hyper::client::pool - pooling idle connection for ("https", www.test.local:8081)
2024-01-18T20:48:28.021028986+11:00 DEBUG e2e_tests::utils - Request to https://www.test.local:8081/endpoint-1 finished after: 369 ms
2024-01-18T20:48:28.021064980+11:00 DEBUG e2e_tests::utils - Response: Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.test.local")), port: Some(8081), path: "/endpoint-1", query: None, fragment: None }, status: 200, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Thu, 18 Jan 2024 09:48:27 GMT"} }
2024-01-18T20:48:28.022376476+11:00 DEBUG rustls::common_state - Sending warning alert CloseNotify
2024-01-18T20:48:28.032818006+11:00 DEBUG hyper::proto::h1::io - parsed 3 headers
2024-01-18T20:48:28.032981786+11:00 DEBUG hyper::proto::h1::conn - incoming body is content-length (10 bytes)
2024-01-18T20:48:28.033086662+11:00 DEBUG hyper::proto::h1::conn - incoming body completed
2024-01-18T20:48:28.033290878+11:00 DEBUG hyper::client::pool - pooling idle connection for ("https", www.test.local:8081)
2024-01-18T20:48:28.033471186+11:00 DEBUG e2e_tests::utils - Request to https://www.test.local:8081/endpoint-1 finished after: 253 ms
2024-01-18T20:48:28.033618652+11:00 DEBUG e2e_tests::utils - Response: Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.test.local")), port: Some(8081), path: "/endpoint-1", query: None, fragment: None }, status: 200, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Thu, 18 Jan 2024 09:48:27 GMT"} }
2024-01-18T20:48:28.034062164+11:00 DEBUG rustls::common_state - Sending warning alert CloseNotify
2024-01-18T20:48:28.067238017+11:00 DEBUG reqwest::connect - starting new connection: https://www.test.local:8081/
2024-01-18T20:48:28.067351010+11:00 DEBUG reqwest::connect - proxy(http://10.12.128.5:3128) intercepts 'https://www.test.local:8081/'
2024-01-18T20:48:28.067384947+11:00 DEBUG hyper::client::connect::http - connecting to 10.12.128.5:3128
2024-01-18T20:48:28.067859496+11:00 DEBUG hyper::client::connect::http - connected to 10.12.128.5:3128
2024-01-18T20:48:28.072120766+11:00 DEBUG rustls::server::hs - decided upon suite TLS13_AES_256_GCM_SHA384
2024-01-18T20:48:28.075779648+11:00 DEBUG hyper::proto::h1::io - flushed 95 bytes
2024-01-18T20:48:28.116652984+11:00 DEBUG hyper::proto::h1::io - parsed 3 headers
2024-01-18T20:48:28.116698112+11:00 DEBUG hyper::proto::h1::conn - incoming body is content-length (10 bytes)
2024-01-18T20:48:28.116718599+11:00 DEBUG hyper::proto::h1::conn - incoming body completed
2024-01-18T20:48:28.116766148+11:00 DEBUG hyper::client::pool - pooling idle connection for ("https", www.test.local:8081)
2024-01-18T20:48:28.116797703+11:00 DEBUG e2e_tests::utils - Request to https://www.test.local:8081/endpoint-9 finished after: 49 ms
2024-01-18T20:48:28.116821464+11:00 DEBUG e2e_tests::utils - Response: Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.test.local")), port: Some(8081), path: "/endpoint-9", query: None, fragment: None }, status: 200, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Thu, 18 Jan 2024 09:48:27 GMT"} }
2024-01-18T20:48:28.117187474+11:00 DEBUG rustls::common_state - Sending warning alert CloseNotify
2024-01-18T20:48:28.232644782+11:00 DEBUG reqwest::connect - starting new connection: https://www.test.local:8081/
2024-01-18T20:48:28.232719668+11:00 DEBUG reqwest::connect - proxy(http://10.12.128.5:3128) intercepts 'https://www.test.local:8081/'
2024-01-18T20:48:28.232757185+11:00 DEBUG hyper::client::connect::http - connecting to 10.12.128.5:3128
2024-01-18T20:48:28.233312092+11:00 DEBUG hyper::client::connect::http - connected to 10.12.128.5:3128
2024-01-18T20:48:28.237308206+11:00 DEBUG rustls::server::hs - decided upon suite TLS13_AES_256_GCM_SHA384
2024-01-18T20:48:28.242369784+11:00 DEBUG hyper::proto::h1::io - flushed 95 bytes
2024-01-18T20:48:28.284908129+11:00 DEBUG hyper::proto::h1::io - parsed 3 headers
2024-01-18T20:48:28.284957148+11:00 DEBUG hyper::proto::h1::conn - incoming body is content-length (10 bytes)
2024-01-18T20:48:28.285122408+11:00 DEBUG hyper::proto::h1::conn - incoming body completed
2024-01-18T20:48:28.285449861+11:00 DEBUG hyper::client::pool - pooling idle connection for ("https", www.test.local:8081)
2024-01-18T20:48:28.285498476+11:00 DEBUG e2e_tests::utils - Request to https://www.test.local:8081/endpoint-0 finished after: 52 ms
Is there a way to achieve that all requests always go through the proxy, and the pooled connection is with the proxy rather than with the final destination?
I came to the conclusion that the behavior was actually expected based on the setup.
Here's a more detailed discussion: https://github.com/seanmonstar/reqwest/issues/2098
I don't think the question I raised here is actually relevant and could be deleted.