How to rebuild apache Livy with scala 2.12

Question

I'm using Spark 3.1.1 which uses Scala 2.12, and the pre-built Livy downloaded from here uses Scala 2.11 (one could find the folder named repl_2.11-jars/ after unzip).

Referred to the comment made by Aliaksandr Sasnouskikh, Livy needs to be rebuilt or it'll throw error {'msg': 'requirement failed: Cannot find Livy REPL jars.'} even in POST Session.

In the README.md, it mentioned:

By default Livy is built against Apache Spark 2.4.5

If I'd like to rebuild Livy, how could I change the spark version that it is built with?

Thanks in advance.

Answer 1

You can rebuild Livy passing spark-3.0 profile in maven to create a custom build for spark 3, for example:

git clone https://github.com/apache/incubator-livy.git && \
cd incubator-livy && \
mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true

This profile is defined in pom.xml, the default one installs Spark 3.0.0. You can change it to use different spark version.

<profile>
      <id>spark-3.0</id>
      <activation>
        <property>
          <name>spark-3.0</name>
        </property>
      </activation>
      <properties>
        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
        <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
        <spark.version>${spark.scala-2.11.version}</spark.version>
        <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
        <netty.spark-2.11.version>4.1.47.Final</netty.spark-2.11.version>
        <netty.version>${netty.spark-2.11.version}</netty.version>
        <java.version>1.8</java.version>
        <py4j.version>0.10.9</py4j.version>
        <json4s.spark-2.11.version>3.5.3</json4s.spark-2.11.version>
        <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
        <json4s.version>${json4s.spark-2.11.version}</json4s.version>
        <spark.bin.download.url>
          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
        </spark.bin.download.url>
        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
      </properties>
    </profile>

As long as I know, Livy supports spark 3.0.x. But worth testing with 3.1.1, and let us know :)

Answer 2

My Spark version is 3.2.1, and my scala version is 2.12.15. I have successfully built and put it into use. I will show my construction process. Pull the master code of livy and modify the pom file as follows：

Finally, execute the package command

mvn clean package -B -V -e -Pspark-3.0 -Pthriftserver -DskipTests -DskipITs -Dmaven.javadoc.skip=true

After Livy's deployment: Test:

Answer 3

I tried to build Livy for Spark 3.1.1 based on rmakoto's answer and it worked! I tinkered a lot and I couldn't exactly remember what I edited in the pom.xml so I am just going to attach my gist link here.

I also had to edit the python-api/pom.xml file to use Python3 to build since there's some syntax error issues when building with the default pom.xml file. Here's the pom.xml gist for python-api.

After that just build with

mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true

Answer 4

Based on @gamberooni 's changes (but using 3.1.2 instead of 3.1.1 for the Spark version and Hadoop 3.2.0 instead of 3.2.1), this is the diff:

diff --git a/pom.xml b/pom.xml
index d2e535a..5c28ee6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -79,12 +79,12 @@
 
   <properties>
     <asynchttpclient.version>2.10.1</asynchttpclient.version>
-    <hadoop.version>2.7.3</hadoop.version>
+    <hadoop.version>3.2.0</hadoop.version>
     <hadoop.scope>compile</hadoop.scope>
     <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
-    <spark.scala-2.12.version>2.4.5</spark.scala-2.12.version>
-    <spark.version>${spark.scala-2.11.version}</spark.version>
-    <hive.version>3.0.0</hive.version>
+    <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
+    <spark.version>${spark.scala-2.12.version}</spark.version>
+    <hive.version>3.1.2</hive.version>
     <commons-codec.version>1.9</commons-codec.version>
     <httpclient.version>4.5.3</httpclient.version>
     <httpcore.version>4.4.4</httpcore.version>
@@ -1060,7 +1060,7 @@
         </property>
       </activation>
       <properties>
-        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
+        <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
         <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
         <spark.version>${spark.scala-2.11.version}</spark.version>
         <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
@@ -1072,9 +1072,9 @@
         <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
         <json4s.version>${json4s.spark-2.11.version}</json4s.version>
         <spark.bin.download.url>
-          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
+          https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
         </spark.bin.download.url>
-        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
+        <spark.bin.name>spark-3.1.2-bin-hadoop3.2</spark.bin.name>
       </properties>
     </profile>
 
diff --git a/python-api/pom.xml b/python-api/pom.xml
index 8e5cdab..a8fb042 100644
--- a/python-api/pom.xml
+++ b/python-api/pom.xml
@@ -46,7 +46,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-            <executable>python</executable>
+            <executable>python3</executable>
               <arguments>
                 <argument>setup.py</argument>
                 <argument>sdist</argument>
@@ -60,7 +60,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-              <executable>python</executable>
+              <executable>python3</executable>
               <skip>${skipTests}</skip>
               <arguments>
                 <argument>setup.py</argument>