When i connect spark Session using Jupyter notebook (using Sparkmagic), Livy. I added venv archives (in spark magic config.json)
in venv, include pyyaml, hdfs library
And i import hdfs in jupyter notebook with pyspark kernel , Error raise no module name.
I already did below link https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html
How can i solve it?
Sorry for that I can't capture the image,
Plus, Let me teach the differences
- spark.pyspark.python
- spark.yarn.appMasterEnv.PYSPARK_PYTHON
- spark.executorEnv.PYSPARK_PYTHON
solve env for pyspark