Failure to run shell script via subprocess.Popen

Question

I am trying to a script run.sh using python subprocess

The script creates a csv file and a state.json file

This is the script(works fine when i run it from terminal) :

#!/bin/bash
export GOOGLE_APPLICATION_CREDENTIALS="/home/anti/Documents/bq_s3/BQ/bin/client_secret.json"

/home/anti/Documents/bq_s3/BQ/bin/tap-bigquery -c /home/anti/Documents/bq_s3/BQ/bin/tap_config.json --catalog /home/anti/Documents/bq_s3/BQ/bin/catalog.json --start_datetime '2013-01-01T00:00:00Z' --end_datetime '2013-01-02T00:00:00Z' | /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target-csv --config /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target_config.json > /home/anti/Documents/bq_s3/state.json

But when i try to run the same from python, it shows the correct output , but doesnt create csv file and a state.json file

This is the python code

#!/usr/bin/python3

import subprocess   
from ast import literal_eval
import io
import os
from subprocess import call

shellscript = subprocess.Popen(["/home/anti/Documents/bq_s3/run.sh"], stdout=subprocess.PIPE)
shellscript.wait()
print (shellscript.returncode)

Output

INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
INFO Running query:
    SELECT start_date,trip_seconds,trip_miles FROM `testbq-305217.test_1.tablename` WHERE 1=1 AND datetime '2013-01-01 00:00:00.000000' <= CAST(start_date as datetime) AND CAST(start_date as datetime) < datetime '2013-01-02 00:00:00.000000' ORDER BY start_date LIMIT 100
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"endpoint": "test_bq"}}
INFO Syncing stream:test_bq
0

The return code is 0, the program works but the csv and json files are not being created when ran from python.

I have given necessary permission chmod u+rx run.sh to the script too.

Answer 1

As you mentioned in the comments since the issue was that current working directory is not set properly pass cwd param to the subprocess.Popen. Better yet, replace subprocess.Popen() with a higher-level function, as recommended in its documentation.

#!/usr/bin/python3

import subprocess   

shellscript = subprocess.call(["/home/anti/Documents/bq_s3/run.sh"], cwd=YOUR_WORKING_DIR)
print(shellscript)