I have one pyspark script that fails sometimes with following error. The script is submitted as YARN with cluster mode.
Application application_1704800602555_46873 failed 2 times due to AM Container for appattempt_1704800602555_46873_000002 exited with exitCode: -1000
Failing this attempt Diagnostics: [2024-03-18 19:06:48.480]
Failed to download resource ( {hdfs://hdfserver/user/user01/sparkStaging/application_1704800602555_46873/file_event.json, 1710774403823, FILE, null },pending, (container_e258_1704800602555_46873_02_000001)],58515885903728016,DOWNLOADING}
java.io.10Exception: Resource hdfs://hdfserver/user/user01/sparkStaging/application_1704800602555_46873/file_event.json is not publicly accessible and as such cannot be part of the public cache.
Any idea what can be the root cause of the issue and how can I resolve it ? How to DEBUG spark-submit execution to see if there is any failure uploading the files to HDFS ?