asyncio loops: how to implement asynio in an existing python program - and share variables/data?

Question

My application needs remote control over SSH. I wish to use this example: https://asyncssh.readthedocs.io/en/latest/#simple-server-with-input

The original app is rather big, using GPIO and 600lines of code, 10 libraries. so I've made a simple example here:

import asyncio, asyncssh, sys, time
# here would be 10 libraries in the original 600line application
is_open = True
return_value = 0;

async def handle_client(process):
    process.stdout.write('Enter numbers one per line, or EOF when done:\n')
    process.stdout.write(is_open)

    total = 0

    try:
        async for line in process.stdin:
            line = line.rstrip('\n')
            if line:
                try:
                    total += int(line)
                except ValueError:
                    process.stderr.write('Invalid number: %s\n' % line)
    except asyncssh.BreakReceived:
        pass

    process.stdout.write('Total = %s\n' % total)
    process.exit(0)

async def start_server():
    await asyncssh.listen('', 8022, server_host_keys=['key'],
                          authorized_client_keys='key.pub',
                          process_factory=handle_client)

loop = asyncio.get_event_loop()

try:
    loop.run_until_complete(start_server())
except (OSError, asyncssh.Error) as exc:
    sys.exit('Error starting server: ' + str(exc))

loop.run_forever()


# here is the "old" program: that would not run now as loop.run_forever() runs.
#while True:
# print(return_value)
# time.sleep(0.1)

The main app is mostly driven by a while True loop with lots of functions and sleep. I've commented that part out in the simple example above.

My question is: How should I implement the SSH part, that uses loop.run_forever() - and still be able to run my main loop? Also: the handle_client(process) - must be able to interact with variables in the main program. (read/write)

Answer 1

You have basically three options:

Rewrite your main loop to be asyncio compatible

A main while True loop with lots of sleeps is exactly the kind of code you want to write asynchronously. Convert this:

while True:
     task_1() # takes n ms
     sleep(0.2)
     task_2() # takes n ms
     sleep(0.4)

into this:

async def task_1():
    while True:
        stuff()
        await asyncio.sleep(0.6)

async def task_2():
    while True:
        stuff()
        await asyncio.sleep(0.01)
        other_stuff()
        await asyncio.sleep(0.8)

loop = asyncio.get_event_loop()
loop.add_task(task_1())
loop.add_task(task_2())
...
loop.run_forever()

This is the most work, but it is almost certain that your current code will be better written, clearer, easier to maintain and easier to develop if written as a bunch of coroutines. If you do this the problem goes away: with cooperative multitasking you tell the code when to yield, so sharing state is generally pretty easy. By not awaiting anything in between getting and using a state var you prevent race conditions: no need for any kind of thread-safe var.

Run your asyncio loop in a thread

Leave your current loop intact, but run your ascynio loop in a thread (or process) with either threading or multiprocessing. Expose some kind of thread-safe variable to allow the background thread to change state, or transition to a (thread safe) messaging paradigm, where the ssh thread emits messages into a queue which your main loop handles in its own time (a message could be something like ("a", 5) which would be handled by doing something like state_dict[msg[0]] == msg[1] for everything in the queue).

If you want to go this way, have a look at the multiprocessing and/or threading docs for examples of the right ways to pass variables or messages between threads. Note that this version will likely be less performant than a pure asyncio solution, particularly if your code is mostly sleeping in the main loop anyhow.

Run your synchronous code in a thread, and have asyncio in the foreground

As @MisterMiyagi points out, asyncio has loop.run_in_executor() for launching a process to run blocking code. It's more generally used to run the odd blocking bit of code without tying up the whole loop, but you can run your whole main loop in it. The same concerns about some kind of thread safe variable or message sharing apply. This has the advantage (as @MisterMiyagi points out) of keeping asyncio where it expects to be. I have a few projects which use background asyncio threads in generally non-asyncio code (event-driven gui code with an asyncio thread interacting with custom hardware over usb). It can be done, but you do have to be careful as to how you write it.

Note btw that if you do decide to use multiple threads, message-passing (with a queue) is usually easier than directly sharing variables.