Here is my code and pom.xml and error can any one figure what is the exact reason .
Code:
def main(args:Array[String]){
val objConf = new SparkConf().setAppName("Spark Connection").setMaster("spark://10.40.10.80:7077")
var sc = new SparkContext(objConf)
val objHiveContext = new HiveContext(sc)
objHiveContext.sql("USE test")
var test= objHiveContext.sql("show tables")
var i =0
var testing = test.collect()
for(i<-0 until testing.length){
println(testing(i))
}
pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
Error Console:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/01/22 18:05:37 INFO SparkContext: Running Spark version 1.5.2
16/01/22 18:05:38 INFO SecurityManager: Changing view acls to: psudhir
16/01/22 18:05:38 INFO SecurityManager: Changing modify acls to: psudhir
16/01/22 18:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(psudhir); users with modify permissions: Set(psudhir)
16/01/22 18:05:41 INFO Slf4jLogger: Slf4jLogger started
16/01/22 18:05:41 INFO Remoting: Starting remoting
16/01/22 18:05:41 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:64657]
16/01/22 18:05:41 INFO Utils: Successfully started service 'sparkDriver' on port 64657.
16/01/22 18:05:41 INFO SparkEnv: Registering MapOutputTracker
16/01/22 18:05:41 INFO SparkEnv: Registering BlockManagerMaster
16/01/22 18:05:42 INFO DiskBlockManager: Created local directory at C:\Users\psudhir\AppData\Local\Temp\blockmgr-2ed0d89c-f370-47bb-8e99-181212bff9c4
16/01/22 18:05:42 INFO MemoryStore: MemoryStore started with capacity 245.7 MB
16/01/22 18:05:42 INFO HttpFileServer: HTTP File server directory is C:\Users\psudhir\AppData\Local\Temp\spark-fddc211b-840f-4a5d-927e-2c7b5c96783b\httpd-44ceeedb-7e5e-430d-bc4d-d35f6e676703
16/01/22 18:05:42 INFO HttpServer: Starting HTTP Server
16/01/22 18:05:42 INFO Utils: Successfully started service 'HTTP file server' on port 64658.
16/01/22 18:05:42 INFO SparkEnv: Registering OutputCommitCoordinator
16/01/22 18:05:43 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/01/22 18:05:43 INFO SparkUI: Started SparkUI at http://172.16.101.215:4040
16/01/22 18:05:43 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/01/22 18:05:43 INFO AppClient$ClientEndpoint: Connecting to master spark://10.40.10.80:7077...
16/01/22 18:05:45 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160122063401-0088
16/01/22 18:05:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64679.
16/01/22 18:05:45 INFO NettyBlockTransferService: Server created on 64679
16/01/22 18:05:45 INFO BlockManagerMaster: Trying to register BlockManager
16/01/22 18:05:45 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.101.215:64679 with 245.7 MB RAM, BlockManagerId(driver, 172.16.101.215, 64679)
16/01/22 18:05:45 INFO BlockManagerMaster: Registered BlockManager
16/01/22 18:05:46 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/01/22 18:05:47 INFO HiveContext: Initializing execution hive, version 1.2.1
16/01/22 18:05:47 INFO ClientWrapper: Inspected Hadoop version: 2.2.0
16/01/22 18:05:47 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0
16/01/22 18:05:47 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/01/22 18:05:47 INFO deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/01/22 18:05:47 INFO deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/01/22 18:05:47 INFO deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/01/22 18:05:47 INFO deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/01/22 18:05:47 INFO deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/01/22 18:05:48 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/01/22 18:05:48 INFO ObjectStore: ObjectStore, initialize called
16/01/22 18:05:48 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/01/22 18:05:48 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/01/22 18:06:00 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/01/22 18:06:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:01 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:08 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/01/22 18:06:08 INFO ObjectStore: Initialized ObjectStore
16/01/22 18:06:08 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/01/22 18:06:08 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/01/22 18:06:09 WARN : Your hostname, psudhir resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:ac10:65d7%13, but we couldn't find any external IP address!
16/01/22 18:06:10 INFO HiveMetaStore: Added admin role in metastore
16/01/22 18:06:10 INFO HiveMetaStore: Added public role in metastore
16/01/22 18:06:11 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/01/22 18:06:11 INFO HiveMetaStore: 0: get_all_databases
16/01/22 18:06:11 INFO audit: ugi=psudhir ip=unknown-ip-addr cmd=get_all_databases
16/01/22 18:06:11 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/01/22 18:06:11 INFO audit: ugi=psudhir ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/01/22 18:06:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/01/22 18:06:15 INFO SessionState: Created local directory: C:/Users/psudhir/AppData/Local/Temp/eb66b3f1-a276-4dd4-8951-90c59d10e3ee_resources
16/01/22 18:06:15 INFO SessionState: Created HDFS directory: /tmp/hive/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee
16/01/22 18:06:15 INFO SessionState: Created local directory: C:/Users/psudhir/AppData/Local/Temp/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee
16/01/22 18:06:15 INFO SessionState: Created HDFS directory: /tmp/hive/psudhir/eb66b3f1-a276-4dd4-8951-90c59d10e3ee/_tmp_space.db
16/01/22 18:06:16 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/01/22 18:06:16 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/01/22 18:06:16 INFO ClientWrapper: Inspected Hadoop version: 2.2.0
16/01/22 18:06:16 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.2.0
16/01/22 18:06:16 INFO deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/01/22 18:06:16 INFO deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/01/22 18:06:16 INFO deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/01/22 18:06:16 INFO deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/01/22 18:06:16 INFO deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/01/22 18:06:16 INFO deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/01/22 18:06:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/22 18:06:17 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/01/22 18:06:17 INFO ObjectStore: ObjectStore, initialize called
16/01/22 18:06:17 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/01/22 18:06:17 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
Exception in thread "main"