I have tried to run pio train
command in HDInsight Spark cluster using following command
pio train -- --deploy-mode cluster --master yarn
But there is following error have been provided
2018-11-05 11:40:05 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: No FileSystem for scheme: wasb
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:121)
at org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:121)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:121)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-11-05 11:40:07 INFO ShutdownHookManager:54 - Shutdown hook called
I use following script to test connection and there is no issue, script successful connect and return available items from Azure Storage
hadoop fs -ls wasb://my_container_name@my_blob_account_name.blob.core.windows.net
Has anyone ideas with solution of the issue?
Had the same issues, where hadoop would support the wasb:// protocole, but not pio According to https://github.com/hning86/articles/blob/master/hadoopAndWasb.md You have to use the hadoop-azure-2.7.1.jar and azure-storage-2.0.0.jar in your CLASSPATH
To solve this issue, you need to add the two jars to the CLASSPATH of pio itself.
With PredictionIO 0.13.1, according to /usr/local/pio/bin/compute-classpath.sh, this can be acheived by adding the jar to the subdirectory plugins
ls /usr/local/pio/plugins/azure-storage-2.0.0.jar ls /usr/local/pio/plugins/hadoop-azure-2.7.1.jar