Writing to file in a Popen subprocess

Question

I am attempting to log the simultaneous serial outputs of 3 GPS receivers to compare their performance on a single computer.

Following the structure outlined in this post, I created a main file which takes in inputs about the current trial for the file naming convention, then POpens a subprocess for each port:

import time
from datetime import datetime
from subprocess import Popen, PIPE


save_dir = "output_csvs/"

sparkfun_port = "COM7"
sparkfun_baud = "38400"

trimble_port = "COM6"
trimble_baud = "38400"

duro_port = "COM5"
duro_baud = "115200"

if __name__ == "__main__":
    # take input to generate file prefix
    file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")
    # 180 for 3 min, 300 for 5 min
    trial_length = input("How long is this trial (in min)? ")
    # VSS6037L (Tallysman Surface Mount)
    # M8HCT (Maxtena)
    # VSP6037L (Tallysman Marine)
    # HC977XF (Tallysman helical)
    # GPS500 (Swift)
    # Zephyr (Trimble)
    antenna = input("Which GPS antenna is being used? ")
    file_prefix += "_" + antenna + trial_length + "min"

    # create filepath for each reciever
    sparkfun_path = save_dir + file_prefix + "_sparkfun.csv"
    trimble_path = save_dir + file_prefix + "_trimble.csv"
    duro_path = save_dir + file_prefix + "_duro.csv"

    # Popen subprocess for each reciever
    sparkfun = Popen(['python', './swap-c_ReadCOM.py', sparkfun_port, sparkfun_baud, sparkfun_path],
                     stdin=PIPE, stdout=PIPE, stderr=PIPE)
    trimble = Popen(['python', './swap-c_ReadCOM.py', trimble_port, trimble_baud, trimble_path],
                    stdin=PIPE, stdout=PIPE, stderr=PIPE)  
    duro = Popen(['python', './swap-c_ReadCOM.py', duro_port, duro_baud, duro_path], stdin=PIPE, stdout=PIPE,
                    stderr=PIPE)  

    # sleep for trial length (+ some margin to be trimmed) then close
    time.sleep(int(trial_length)*60+1)
    print("Trial Complete")
    quit()

Then, I created the subprocess file swap-c_ReadCOM.py, which is responsible for opening the specified COM port, listening to it, filtering for only GGA nmea strings, and writing said strings to a csv file.

swap-c_ReadCOM.py

import sys
import serial
import re
import csv


def trim_checksum(decoded_str):
    idx = decoded_str.find('*')
    if idx != -1:
        return decoded_str[:idx]
    return decoded_str


filepath = str(sys.argv[3])
ser = serial.Serial(port=sys.argv[1], baudrate=int(sys.argv[2]), bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE)

while True:  # The program never ends... will be killed when master is over.
    ser_bytes = ser.readline()
    decoded_bytes = ser_bytes[0:len(ser_bytes) - 2].decode("utf-8")
    print(decoded_bytes)
    isGGA = re.search("\$\w\wGGA", decoded_bytes)
    if isGGA is not None:
        decoded_bytes = trim_checksum(decoded_bytes)
        with open(filepath, "a", newline='') as f:
            split = decoded_bytes.split(",")
            writer = csv.writer(f)
            writer.writerow(split)

To test this code, I began by attempting to only run one subprocess, commenting out the others. The main file runs to completion, but no csv is generated. Should I be piping the serial input back to main, to write to a file from there?

After modifying the main to print the output of my subprocess, it seems that swap-c_ReadCOM.py doesn't capture the serial input when run as a subprocess, as my code simply printed b' '. The csv file wouldn't be created then, as the regex is never cleared. When running swap-c_ReadCOM.py from the command line, the serial input is captured correctly and the csv file is created.

Answer 1

I can think of two possible explanations.

I created mock scripts called main.py and worker.py. Notice save_dir = "A_Place_With_No_Name/".

1. This code(main.py) run but doesn't create files. Because save_dir and ./worker.py are relative paths, your code won't write a file when you run the program in another directory or the wrong directory name of save_dir.

Also, in this case, your subprocesses died early because of FileNotFoundError: [Errno 2] No such file or directory:

So, change save_dir = "output_csvs/" and ./swap-c_ReadCOM.py to an absolute path, or if there is no save_dir in the directory, create it.

2. If your system is Windows, according to @jasonharper, you can't create file name with colon(file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")). e.g., I tested with open('colon:colon.txt, 'a') with a string to write. It created empty colon file in Windows 10. So avoid ":" in file_prefix.

main.py

import time
from datetime import datetime
from subprocess import Popen, PIPE

# This is proof that the subprocess dies early without writing files.
save_dir = "A_Place_With_No_Name/"

sparkfun_port = "COM7"
sparkfun_baud = "38400"

trimble_port = "COM6"
trimble_baud = "38400"

duro_port = "COM5"
duro_baud = "115200"

if __name__ == "__main__":

    file_prefix = "" + datetime.now().strftime("%m-%d-%Y-%H:%M:%S")
    trial_length = input("How long is this trial (in min)? ")
    antenna = input("Which GPS antenna is being used? ")
    file_prefix += "_" + antenna + trial_length + "min"

    sparkfun_path = save_dir + file_prefix + "_sparkfun.csv"
    trimble_path = save_dir + file_prefix + "_trimble.csv"
    duro_path = save_dir + file_prefix + "_duro.csv"

    # Popen subprocess for each reciever
    sparkfun = Popen(['python', './worker.py', sparkfun_port, sparkfun_path],
                     stdin=PIPE, stdout=PIPE, stderr=PIPE)
    trimble = Popen(['python', './worker.py', trimble_port, trimble_path],
                    stdin=PIPE, stdout=PIPE, stderr=PIPE)  
    duro = Popen(['python', './worker.py', duro_port, duro_path], stdin=PIPE, stdout=PIPE,
                    stderr=PIPE)  

    # sleep for trial length (+ some margin to be trimmed) then close
    time.sleep(int(trial_length)*60+1)
    print("Trial Complete")
    quit()

worker.py

import sys
import time
import re
import csv


filepath = str(sys.argv[2])

while True:  # The program never ends... will be killed when master is over.
    print(sys.argv[1:])
    # FileNotFoundError: [Errno 2] No such file or directory:
    with open(filepath, "a", newline='') as f:
        split = sys.argv[1:]
        writer = csv.writer(f)
        writer.writerow(split)
    time.sleep(1)