Is "User-Agent" header mandatory in https GET request as per RFC

Question

I was trying to download a file from a server using mbedtls. used the following GET request

GET /xxxxxxxxxxxxxxx.xml HTTP/1.1

Host: example_xxxxx.com

Connection: keep-alive

The server responded with status 403

When I add a "User-Agent" header filed, it started working.

Is User-Agent mandatory for HTTP GET requests as per RFC specification.

Answer 1

RFC 2616 14.43:

User agents SHOULD include this field with requests.

As far as the RFC is concerned, it's not required.

The server you're accessing has chosen to require the user agent, possibly as some rudimentary anti-scraping mechanism.