Passing multiple jar files in dcos spark-submit, jars with comma separated not suitable

Question

uggestions needed, need to pass lots of jar files to dcos spark submit, jars with comma separated not suitable:

Tried below options:

dcos spark run --submit-args='--class com.gre.music.inn.orrd.SpaneBasicApp --jars /spark_submit_jobs/new1/unzip_new/* 30'

dcos spark run --submit-args='--class com.gre.music.inn.orrd.SpaneBasicApp --jars local:* 30'

dcos spark run --submit-args='--class com.gre.music.inn.orrd.SpaneBasicApp --jars https://s3-us-west-2.amazonaws.com/gmu_jars/* 30‘ . 

The last one wont work bcz I guess wild card is not allowed with http.

Answer 1

As far as I know you can't use wildcards, and you need to put the JARs somewhere where Spark can access them in a distributed manner (S3, http, hdfs, etc.).

See

• http://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit

application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.

Answer 2

Update from DC/OS:

--jars isn't supported via dcos spark run (Spark cluster mode). We'll have support for it around ~ DC/OS 1.10 when we move Spark over to Marathon instead of the Spark dispatcher. In the mean time, if you want to use --jars, you'll have to submit your job in client mode via spark-submit through metronome or marathon.

Answer 3

You can't use wildcards with --jars argument in spark-submit. Here's the feature request for that (it's still open).