How to get Python in Qubole to save CSV and TXT files to Azure data lake?

Question

I have Qubole connected to Azure data lake, and I can start a spark cluster, and run PySpark on it. However, I can't save any native Python output, like text files or CSVs. I can't save anything other than Spark SQL DataFrames. What should I do to resolve this? Thank you in advance!

Answer 1

If I understand your question correctly,I believe you are unable to download the result of pyspark command output into text or CSVs while you are able to do so for the spark sql command command output in a nice tabular format.

Unfortunately, there is no direct field separator for the output text for a Python or Shell command outputs. You will need to get your output comma separated so you can download the raw output and save it as a csv.

If this is not what you meant, Please share more details as to what exactly are you trying to do along with screenshots details. As that will help us answer your question better.

Answer 2

I resolved it. I needed to add the file to the PySpark session using textFile() details and sample code here For any file I want, I need to add it to the spark session. For example, if I needed to add a .py file from Azure data lake, I need to add it using addPyFile() with the path to the file.