Apache Spark + Mobius Debug mode

212 Views Asked by At

I am new to Apache spark and trying to run this on Windows machine with Mobius. but i am getting exception while running it. first of all i have setup following environment variables

JAVA_HOME C:\Program Files\Java\jdk1.8.0_152
HADOOP_HOME C:\winutils
SPARK_HOME  c:\spark [spark-2.0.2-bin-hadoop2.6]
SPARKCLR_HOME E:\SparkClr\spark-clr_2.11-2.0.200\runtime

In code provided correct file to read data from as file:///E:/Mobius/Mobius/SampleText.txt

I tried to run it in debug mode

E:\SparkClr\spark-clr_2.11-2.0.200\runtime\scripts>sparkclr-submit.cmd debug
[sparkclr-submit.cmd] SPARKCLR_JAR=spark-clr_2.11-2.0.200.jar
[sparkclr-submit.cmd] LAUNCH_CLASSPATH="E:\SparkClr\spark-clr_2.11-2.0.200\runtime\lib\spark-clr_2.11-2.0.200.jar;c:\spark\jars\*"
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/12/19 11:34:23 INFO CSharpRunner: Starting CSharpBackend!
17/12/19 11:34:25 INFO CSharpRunner: Port number used by CSharpBackend is 5567
***********************************************************************
* [CSharpRunner.main] Backend running debug mode. Press enter to exit *
***********************************************************************
17/12/19 11:35:09 INFO SparkContext: Running Spark version 2.0.2

In my App.config Added following Entry [Port name with i got from above]

<add key="CSharpBackendPortNumber" value="5567"/>

I am getting below Exception message

    JVM method execution failed: Nonstatic method partitions failed for class 12 when called with no parameters

Stack Trace
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName)
   at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.GetNumPartitions()
   at Microsoft.Spark.CSharp.Core.RDD`1.GetDefaultPartitionNum()
   at Microsoft.Spark.CSharp.Core.PairRDDFunctions.PartitionBy[K,V](RDD`1 self, Int32 numPartitions, Func`2 partitionFunc)
   at Microsoft.Spark.CSharp.Core.PairRDDFunctions.ReduceByKey[K,V](RDD`1 self, Func`3 reduceFunc, Int32 numPartitions)
   at Microsoft.Spark.CSharp.Examples.WordCountExample.Main(String[] args) in E:\Mobius\Mobius\examples\Batch\WordCount\Program.cs:line 35

I am not sure about CSharpWorkerPath.

Please help if anyone face this kind of issue earlier.

0

There are 0 best solutions below