Im trying to load a large data set stored in parquet format to elastic using pyspark and the script exits with the following error. Im very new to this and would like a direction on resolving this.
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO ShutdownHookManager: Deleting directory /var/data/spark-{some value}
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO ShutdownHookManager: Deleting directory /tmp/spark-{some value}
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO ShutdownHookManager: Deleting directory /var/data/spark-{some value}
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO ShutdownHookManager: Shutdown hook called
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO SparkContext: Successfully stopped SparkContext
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO BlockManagerMaster: BlockManagerMaster stopped
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO BlockManager: BlockManager stopped
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO MemoryStore: MemoryStore cleared
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
Mar 28 14:02:47.969
Mar 28 14:02:47.969
warnings.warn(
Mar 28 14:02:47.969
Mar 28 14:02:47.969
/home/sparkuser/.local/lib/python3.8/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'some.url'. Adding certificate verification is strongly advised. See: https://url.com
Mar 28 14:02:47.969
Mar 28 14:02:47.969
Delete partial index
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO DAGScheduler: ResultStage 1 (runJob at EsSparkSQL.scala:103) failed in 6627.758 s due to Stage cancelled because SparkContext was shut down
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO DAGScheduler: Job 1 failed: runJob at EsSparkSQL.scala:103, took 6627.862084 s
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:47 INFO SparkUI: Stopped Spark web UI at http:some-url:4040
Mar 28 14:02:47.969
Mar 28 14:02:47.969
24/03/28 08:32:46 INFO SparkContext: SparkContext is stopping with exitCode 0.
Mar 28 14:02:47.968
Mar 28 14:02:47.968
24/03/28 08:32:46 INFO SparkContext: Invoking stop() from shutdown hook
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Shuffle files lost for executor: 4 (epoch 2)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMaster: Removed 4 successfully in removeExecutor
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(4, {some ip}, {some port}, None)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: No executor found for {some ip}:{some port}
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Executor lost: 4 (epoch 2)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Disabling executor 4.
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Shuffle files lost for executor: 12 (epoch 1)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMaster: Removed 12 successfully in removeExecutor
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(12, {some ip}, {some port}, None)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: No executor found for {some ip}:{some port}
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Trying to remove executor 12 from BlockManagerMaster.
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Executor lost: 12 (epoch 1)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Disabling executor 12.
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Shuffle files lost for executor: 10 (epoch 0)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMaster: Removed 10 successfully in removeExecutor
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(10, {some ip}, {some port}, None)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster.
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO DAGScheduler: Executor lost: 10 (epoch 0)
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: No executor found for {some ip}:{some port}
Mar 28 14:02:46.968
Mar 28 14:02:46.968
24/03/28 08:32:46 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Disabling executor 10.