Spark Deployment in Azure cloud

Question

Is it possible to deploy spark code in Azure cloud without the yarn component? thanks in advance

Answer 1

I don't think it is possible to deploy HDInsight cluster without YARN.Refer to the HDInsight documentation

• https://learn.microsoft.com/en-sg/azure/hdinsight/hdinsight-hadoop-introduction
• https://learn.microsoft.com/en-sg/azure/hdinsight/hdinsight-component-versioning

YARN is the resource manager for Hadoop. Is there any particular reason you would not want to use YARN while working with HDInsight Spark cluster?

If you want to use the standalone mode, you can modify the location of the master url while submitting the job using Spark-submit command.

I have some examples in my repo with Spark-submit both in local mode and on HDInsight cluster https://github.com/NileshGule/learning-spark

You can refer to

• local mode : https://github.com/NileshGule/learning-spark/blob/master/src/main/java/com/nileshgule/movielens/MovieLens.md
• HDInsight Spark cluster : https://github.com/NileshGule/learning-spark/blob/master/Azure.md

Answer 2

Yes,you can deploy Apache Spark cluster in Azure HDInsight without Yarn.

Spark clusters in HDInsight include the following components that are available on the clusters by default.

1)Spark Core. Includes Spark Core, Spark SQL, Spark streaming APIs, GraphX, and MLlib.

2)Anaconda

3)Livy

4)Jupyter notebook

5)Zeppelin notebook

Spark clusters on HDInsight also provide an ODBC driver for connectivity to Spark clusters in HDInsight from BI tools such as Microsoft Power BI and Tableau.

Refer to the following sites for more information,

Create an Apache Spark cluster in Azure HDInsight

Introduction to Spark on HDInsight