aiohttp with connector. Session is closed & Unclosed connector excepions

Question

My program send requests to the site several times via aiohttp. I also need to use a proxy, so I use a connector from the aiohttp-socks library. But after implementing the functionality, I encountered an exception.

I made an illustrative example that can be used to reproduce the exception.

requirements.txt

aiohttp==3.8.4
aiohttp-socks==0.8.0
PySocks==1.7.1

main.py

import asyncio
from typing import Optional

import aiohttp
from aiohttp_socks import ProxyConnector


async def async_get(url: str, connector: Optional[ProxyConnector] = None, **kwargs) -> str:
    async with aiohttp.ClientSession(connector=connector) as session:
        print(session)
        async with session.get(url=url, **kwargs) as response:
            status_code = response.status
            if status_code <= 201:
                return await response.text()

            raise Exception(f'Unsuccessful request: {status_code}')


async def main():
    proxy = 'socks5://username:[email protected]:1111'
    connector = ProxyConnector.from_url(url=proxy, rdns=True)
    for _ in range(3):
        await async_get(url='https://www.google.com/', connector=connector)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output

<aiohttp.client.ClientSession object at 0x0000011A5F468940>
Traceback (most recent call last):
  File "...\program\main.py", line 28, in <module>
    loop.run_until_complete(main())
  File "...\Python\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "...\program\main.py", line 23, in main
    await async_get(url='https://www.google.com/', connector=connector)
  File "...\program\main.py", line 11, in async_get
    async with session.get(url=url, **kwargs) as response:
  File "...\program\venv\lib\site-packages\aiohttp\client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "...\program\venv\lib\site-packages\aiohttp\client.py", line 400, in _request
    raise RuntimeError("Session is closed")
RuntimeError: Session is closed
<aiohttp.client.ClientSession object at 0x0000011A5F48FE50>

After searching other people's solutions, I realized that after closing the session and the connector, it keeps pointing to the same session. In order for this not to happen, I need to do one of the following:

1. Initialize a new connector for each session;
2. Pass the connector_owner=False argument to the session so that it doesn't close the connector.

I decided to use the second method:

    async with aiohttp.ClientSession(connector=connector, connector_owner=False) as session:

But when I ran the program with this fix a new exception appeared:

<aiohttp.client.ClientSession object at 0x0000024DD07D9940>
<aiohttp.client.ClientSession object at 0x0000024DD07FFE50>
<aiohttp.client.ClientSession object at 0x0000024DD07FF550>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x0000024DD08066A0>, 69074.984)]']
connector: <aiohttp_socks.connector.ProxyConnector object at 0x0000024DD07D9820>

Answer 1

After trying unsuccessfully to find some solutions, I decided to figure it out on my own.

Since the exception says that the connector isn't closed, then it's logical that I should close it after the program has finished. I found two ways to do it.

connector.close()

The most obvious way to do this is to use a special function.

main.py

import asyncio
from typing import Optional

import aiohttp
from aiohttp_socks import ProxyConnector


async def async_get(url: str, connector: Optional[ProxyConnector] = None, **kwargs) -> str:
    async with aiohttp.ClientSession(connector=connector, connector_owner=False) as session:
        print(session)
        async with session.get(url=url, **kwargs) as response:
            status_code = response.status
            if status_code <= 201:
                return await response.text()

            raise Exception(f'Unsuccessful request: {status_code}')


async def main():
    proxy = 'socks5://username:[email protected]:1111'
    connector = ProxyConnector.from_url(url=proxy, rdns=True)
    for _ in range(3):
        await async_get(url='https://www.google.com/', connector=connector)

    connector.close()  # Close connector


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

This method is suitable for a simple program like the example. My program is much more complex, so it will be difficult to implement this method. Besides, after several runs I noticed that the print function in some cases prints two identical sessions. You can use print(id(session)) to make sure they are identical.

<aiohttp.client.ClientSession object at 0x0000027BC8ED8940>
<aiohttp.client.ClientSession object at 0x0000027BC8EFFE50>
<aiohttp.client.ClientSession object at 0x0000027BC8EFFE50>

For my program, this behavior is undesirable. So I kept looking for solutions and came up with one more.

force_close=True connector argument

If I want a new session to be used for each request, I need to find how to do that. By exploring the base connector I found the force_close=True argument which does exactly what I need it to do - close connection after request and use a new one for each following request.

Bottom line, the final solution I am using is as follows.

main.py

import asyncio
from typing import Optional

import aiohttp
from aiohttp_socks import ProxyConnector


async def async_get(url: str, connector: Optional[ProxyConnector] = None, **kwargs) -> str:
    async with aiohttp.ClientSession(connector=connector, connector_owner=False) as session:
        print(session)
        async with session.get(url=url, **kwargs) as response:
            status_code = response.status
            if status_code <= 201:
                return await response.text()

            raise Exception(f'Unsuccessful request: {status_code}')


async def main():
    proxy = 'socks5://username:[email protected]:1111'
    connector = ProxyConnector.from_url(url=proxy, rdns=True, force_close=True)
    for _ in range(3):
        await async_get(url='https://www.google.com/', connector=connector)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())