Asyncio "fire and forget" tasks in a separate thread

Question

I have a long-running synchronous Python program, where I'd like to run ~10 "fire and forget" tasks every second. These tasks hit a remote API and do not need to return any value. I tried this answer, but it requires too much CPU/memory to spawn and maintain all the separate threads so I've been looking into asyncio.

This answer explained nicely how to run "fire and forget" using asyncio. However, it requires using run_until_complete(), which waits until all the asyncio tasks are done. My program is using sync Python so this doesn't work for me. Ideally, the code should be as simple as this, where log_remote won't block the loop:

while True:
    latest_state, metrics = expensive_function(latest_state)
    log_remote(metrics) # <-- this should be run as "fire and forget"

I'm on Python 3.7. How can I easily run this using asyncio on another thread?

Answer 1

You can start a single event loop in a single background thread and use it for all your fire&forget tasks. For example:

import asyncio, threading

_loop = None

def fire_and_forget(coro):
    global _loop
    if _loop is None:
        _loop = asyncio.new_event_loop()
        threading.Thread(target=_loop.run_forever, daemon=True).start()
    _loop.call_soon_threadsafe(asyncio.create_task, coro)

With that in place, you can just call fire_and_forget on a coroutine object, created by calling an async def:

# fire_and_forget defined as above

import time

async def long_task(msg):
    print(msg)
    await asyncio.sleep(1)
    print('done', msg)

fire_and_forget(long_task('foo'))
fire_and_forget(long_task('bar'))
print('continuing with something else...')
time.sleep(3)

Note that log_remote will need to actually be written as an async def using asyncio, aiohttp instead of requests, etc.