I would like to use kudu with pyspark. While I can use it with:
sc.read.format('org.apache.kudu.spark.kudu').option('kudu.master',"hdp1:7051").option('kudu.table',"impala::test.z_kudu_tab").load()
I cannot find a way to import KuduContext. I'm working in a jupyter notebook, and importing it with:
os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 2g --packages com.ibm.spss.hive.serde2.xml:hivexmlserde:1.0.5.3 --packages org.apache.kudu:kudu-spark2_2.11:1.7.0 pyspark-shell"
My not working code:
kudu_Context = KuduContext("es2-hdp1:7051", sc)
Dies with error:
NameError: name 'KuduContext' is not defined
I've also tried:
kudu_context = sc._jvm.org.apache.kudu.spark.kudu.KuduContext("hdp1:7051", sc.sparkContext)
which dies with error:
AttributeError: 'SparkContext' object has no attribute '_get_object_id'