how to execute and monitor a long-running process remotely and view output before completion

Question

how can I execute a process on a remote machine and view the standard output before the process exits? ssh is ideal but only reports output once the remote command has completed.

an example python script like this:

#!/usr/bin/python3
import time

tStart = time.time()
elapsedTime=0

while elapsedTime<10000:
    time.sleep(2)
    elapsedTime = time.time() - tStart
    print('elapsed time: {0:6.2f}(s)'.format(elapsedTime))

generates output indicating it's progress:

elapsed time:   2.00(s)
elapsed time:   4.00(s)
elapsed time:   6.00(s)
elapsed time:   8.00(s)
...

I'd like to run this on a remote machine with ssh:

ssh vector /home/comperem/myProc.py

This process will not end for many minutes or hours and I need to monitor the process as it runs.

How can I get ssh or something similar to run the command remotely and report stdout to the local console as it is generated, before the remote process completes? Output after each line would be useful.

Answer 1

ssh vector /home/comperem/myProc.py

When you invoke a remote program through ssh in this fashion, the remote process runs without a TTY by default. Instead, it is connected to the ssh server through a set of pipes. Your python program is probably buffering its output because it's writing to a pipe rather than a TTY.

The simplest fix is for your program to flush its output. This question discusses how to get a python script to flush standard output. I'm not a python programmer, but it looks like this should work:

while elapsedTime<10000:
    time.sleep(2)
    elapsedTime = time.time() - tStart
    print('elapsed time: {0:6.2f}(s)'.format(elapsedTime), flush=True)
                                                         ^^^^^^^^^^^^

The other question discusses other ways to do it.

Answer 2

You can resolve your question with daggy:

1. Create Data Aggregation Config config.yaml like this:

aliases:
 - &myProc
   name: myProc
   command: tail -f /home/comperem/myProc.py
   extension: log
 - &auth_ssh
   login: user
   password: password

sources:
   host1:
     type: ssh
     connection:
       *auth_ssh
     host: 192.168.1.10
     commands:
       - *myProc
    hostN:
     type: ssh
     connection:
       *auth_ssh
     host: 192.168.1.N
     commands:
       - *myProc

2. Run daggy -o my_proc_folder config.yaml
3. Run tail -f my_proc_folder/*