Use AsyncSSH to collect files from multiple computers

Question

I have a program, which collects various log-files from multiple computers into a single ZIP-file on the client machine. It is currently using Fabric and I'd like to rewrite it with AsyncSSH to make all the servers send their files in parallel.

What I'm trying to figure out is getting the files over -- as streams. My current Fabric-based code is:

    result = io.BytesIO()
    connection.get(remote = path, local = result)
    result.seek(0)
    return result

What's the equivalent for when the connection is a result of asyncssh.connect()?

Answer 1

I think you're looking for create_process() instead of run(). That will return you an SSHClientProcess object that has stdin/stdout/stderr classes which are SSHReader and SSHWriter objects that behave similar to asyncio's StreamReader and StreamWriter.

Alternately, if "connection.get" there is actually doing an SFTP file transfer, I think you want to look into use open() on the SFTPClient object, rather than get(). That would give you a file object you could call read() on. With SFTP you normally want to schedule several reads from the same file in parallel to improve speed, though, and so the read() calls take an offset and a size rather than treating it like a stream that you sequentially access.