Specify SNI server_hostname when performing request with asyncio/aiohttp

Question

Hello fellow developers ! I'm stuck in a corner case and I'm starting to be out of hairs to pull... Here is the plot :

            load-balancer.example.com:443 (TCP passthrough)
                               /\
                              /  \
                             /    \
                            /      \
          s1.example.com:443        s2.example.com:443
              (SSL/SNI)                  (SSL/SNI)              

The goal is to stress-test the upstreams s1 and s2 directly using aiohttp with certificate-validation enable. Since the load-balancer does not belong to me I don't want to do the stress-test over it.

• the code is not supposed to run on other platforms than GNU Linux with at least Python-v3.7 (but I can use any recent version if needed)
• all servers serve a valid certificate for load-balancer.example.com
• openssl validates the certificate from the upstreams when using openssl s_connect s1.example.com:443 -servername load-balancer.example.com
• cURL needs curl 'https://load-balancer.example.com/' --resolve s1.example.com:443:load-balancer.example.com and also validates successfully

I am able to launch a huge batch of async ClientSession.get requests on both upstreams in parallel but for each request I need to somehow tell asyncio or aiohttp to use load-balancer.example.com as server_hostname, otherwise the SSL handshake fails.

Is there an easy way to setup the ClientSession to use a specific server_hostname when setting up the SSL socket ? Does someone have already done something like that ?

EDIT : here is the most simple snippet with just a single request :

import aiohttp
import asyncio


async def main_async(host, port, uri, params=[], headers={}, sni_hostname=None):
    if sni_hostname is not None:
        print('Setting SNI server_name field ')
        #
        # THIS IS WHERE I DON'T KNOW HOW TO TELL aiohttp
        # TO SET THE server_name FIELD TO sni_hostname
        # IN THE SSL SOCKET BEFORE PERFORMING THE SSL HANDSHAKE
        #
    try:
        async with aiohttp.ClientSession(raise_for_status=True) as session:
            async with session.get(f'https://{host}:{port}/{uri}', params=params, headers=headers) as r:
                body = await r.read()
                print(body)
    except Exception as e:
        print(f'Exception while requesting ({e}) ')


if __name__ == "__main__":
    asyncio.run(main_async(host='s1.example.com', port=443,
                           uri='/api/some/endpoint',
                           params={'apikey': '0123456789'},
                           headers={'Host': 'load-balancer.example.com'},
                           sni_hostname='load-balancer.example.com'))

When running it with real hosts, it throws

Cannot connect to host s1.example.com:443 ssl:True 
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] '
certificate verify failed: certificate has expired (_ssl.c:1131)')]) 

Note that the error certificate has expired indicates that the certificate proposed to the client is the default certificate since the SNI hostname is s1.example.com which is unknow by the webserver running there.

When running it against the load-balancer it works just fine, the SSL handshake happens with the upstreams which serve the certificate and everything is valid.

Also note that

• sni_callback does not help since it is called after the handshake has started and the certificate was received (and at this point server_hostname is a read-only property anyway)
• it does not seem to be possible to set server_hostname when creating an SSLContext allthough SSLContext.wrap_socket does support server_hostname but I was not able to make that work

I hope someone knows how to fill the comment block in that snippet ;-]

Answer 1

Apologies for the delayed response, but I recently grappled with the same challenge. To my knowledge, neither the popular asynchronous HTTP library aiohttp nor any other contemporary asynchronous HTTP libraries natively support defining the SNI (Server Name Indication) field. To circumvent this limitation, I successfully employed the urllib3 library and implemented a workaround similar to the one you've provided:

sni_proxy_ip_addr = "127.0.0.1"  # your SNI proxy server
server_hostname: str = "ifconfig.co"
server_hostname_uri: str = "/json"

pool = urllib3.HTTPSConnectionPool(
    sni_proxy_ip_addr,
    port=443,
    server_hostname=server_hostname
)

try:
    resp = pool.request(
        "GET",
        server_hostname_uri,
        headers={"Host": server_hostname},
        assert_same_host=False,
        retries=False
    )
except urllib3.exceptions.MaxRetryError:
    return {'original_ip_addr': sni_proxy_ip_addr}
except urllib3.exceptions.NewConnectionError:
    return {'original_ip_addr': sni_proxy_ip_addr}

PS. google bard is amazing to rewrite your response in more formal ways :)))))