I am new to Apache spark and trying to run this on Windows machine with Mobius. but i am getting exception while running it. first of all i have setup following environment variables
JAVA_HOME C:\Program Files\Java\jdk1.8.0_152
HADOOP_HOME C:\winutils
SPARK_HOME c:\spark [spark-2.0.2-bin-hadoop2.6]
SPARKCLR_HOME E:\SparkClr\spark-clr_2.11-2.0.200\runtime
In code provided correct file to read data from as file:///E:/Mobius/Mobius/SampleText.txt
I tried to run it in debug mode
E:\SparkClr\spark-clr_2.11-2.0.200\runtime\scripts>sparkclr-submit.cmd debug
[sparkclr-submit.cmd] SPARKCLR_JAR=spark-clr_2.11-2.0.200.jar
[sparkclr-submit.cmd] LAUNCH_CLASSPATH="E:\SparkClr\spark-clr_2.11-2.0.200\runtime\lib\spark-clr_2.11-2.0.200.jar;c:\spark\jars\*"
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/12/19 11:34:23 INFO CSharpRunner: Starting CSharpBackend!
17/12/19 11:34:25 INFO CSharpRunner: Port number used by CSharpBackend is 5567
***********************************************************************
* [CSharpRunner.main] Backend running debug mode. Press enter to exit *
***********************************************************************
17/12/19 11:35:09 INFO SparkContext: Running Spark version 2.0.2
In my App.config Added following Entry [Port name with i got from above]
<add key="CSharpBackendPortNumber" value="5567"/>
I am getting below Exception message
JVM method execution failed: Nonstatic method partitions failed for class 12 when called with no parameters
Stack Trace
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName)
at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.GetNumPartitions()
at Microsoft.Spark.CSharp.Core.RDD`1.GetDefaultPartitionNum()
at Microsoft.Spark.CSharp.Core.PairRDDFunctions.PartitionBy[K,V](RDD`1 self, Int32 numPartitions, Func`2 partitionFunc)
at Microsoft.Spark.CSharp.Core.PairRDDFunctions.ReduceByKey[K,V](RDD`1 self, Func`3 reduceFunc, Int32 numPartitions)
at Microsoft.Spark.CSharp.Examples.WordCountExample.Main(String[] args) in E:\Mobius\Mobius\examples\Batch\WordCount\Program.cs:line 35
I am not sure about CSharpWorkerPath.
Please help if anyone face this kind of issue earlier.