im comparing between pyspark local mode and standalone mode where
local :
findspark.init('C:\spark\spark-3.0.3-bin-hadoop2.7')
conf=SparkConf()
conf.setMaster("local[*]")
conf.setAppName('firstapp')
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
standalone :
findspark.init('C:\spark\spark-3.0.3-bin-hadoop2.7')
conf=SparkConf()
conf.setMaster("spark://127.0.0.2:7077")
conf.setAppName('firstapp')
sc = SparkContext(conf=conf)
spark = SparkSession(sc)
plus starting the Master and the workers using :
Master
bin\spark-class2.cmd org.apache.spark.deploy.master.Master
Worker multiple times depending on the number of workers
bin\spark-class2.cmd org.apache.spark.deploy.worker.Worker -c 1 -m 1G spark://127.0.0.1:7077 where '1' mean one core and '1G' mean 1gb or Ram.
my question is : what is the difference between local mode and standalone mode in term of the usage of threads and cores ?