I'm running beeline.sh through a python script and reading its output after submitting numerous sequential SQL queries to be executed through it. I can't send all queries in a single run, I have to run each of these queries and inspect their effect individually.
Unfortunately I can't reliably decide if a query was fully processed or not in order to inspect the output and then send another query to proceed with the purpose of my script.
I thought I can wait for the prompt to be printed because that's what I would see after manually running a query through beeline.sh, but the prompt wouldn't be captured while I'm reading the process's stdout line by line!
process = subprocess.Popen(['bash', self.beeline_path], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
line = self.process.stdout.readline().decode("utf-8").strip()
while line is not None and len(line) > 0:
if line.startswith("0: jdbc:hive2://"):
logging.info("Found prompt: " + line) # Never reached here!
break
else:
line = self.process.stdout.readline().decode("utf-8").strip()
So is there a way to make sure the prompt is captured or at least a reliable way to decide when a query was completely processed?