I am using Page.route to intercept all requests, and add the auth header to only specific request URLs. I decided to test to see if there was a performance impact on how long it took the page to get to "networkidle" between setting the header indiscriminately (ie page.set_extra_http_headers) and intercepting each request before it was sent out. To my surprise, I found that intercepting requests before they were sent was measurably faster on real web pages, and drastically faster in some other scenarios:
type=request interception (passthrough) n= 30 std_dev=0.07077 mean=2.66928
type=normal n= 30 std_dev=0.24912 mean=12.10215
Opening my pathological URL in my own installation of Chrome, I observed a time right in the the range of type=normal. I'm wondering why this is. Chrome seems to be throttling the number of concurrent requests. The same page in Firefox reaches a network idle state very quickly. I observed this with Playwright 1.40.0 (latest), as well as some earlier versions.
This is how I'm routing the requests:
async def passthrough(route: Route, req: Request):
await route.continue_()
await page.route("*", passthrough)
Full test code here
Test server code:
import asyncio
import random
from flask import Flask
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
@app.route('/ping', methods=['GET'])
async def ping():
# if omitted, there is still a measurable difference, but it's not as large
await asyncio.sleep(random.random() / 25)
return "pong"
if __name__ == '__main__':
app.run(debug=True)
Test page HTML:
<!DOCTYPE html>
<html lang="en">
<body>
<script>
function sendRequests() {
for (let i = 1; i <= 500; i++) {
fetch('http://127.0.0.1:5000/ping', { method: 'GET' })
.then(response => console.log(`Request ${i}: ${response.status}`))
.catch(error => console.error(`Request ${i}: ${error.message}`));
}
}
sendRequests();
</script>
</body>
</html>