Access a data file from the current livy session

Question

I have a Spark Cluster running on Hadoop in YARN mode. I have configured Livy server to interact and submit client spark jobs to the spark cluster. I uploaded a data file along with the jar from the java program to Livy which gets uploaded in the .livySessions folder in HDFS.

LivyClient client = new LivyClientBuilder(true).setURI(new URI("http://191.111.123.11:8998")).build();
client.uploadFile(new File("/C:/HAL/kmeans_data.txt")).get();
Object str = client.uploadJar(new File("/C:/Users/msingh/livy-spring-app-1.3.6.RELEASE.jar")).get();

How should I configure my program to read the data file from the current Livy session for execution.

Answer 1

You could use SparkFiles.get () which will fetch these files from within your spark job. Try and let me know.