Java KCL SPark Streaming can't import org.apache.spark.streaming.kinesis

132 Views Asked by At

Hello guys I'm trying to perform a KCL within a Streaming Spark job, I have created a first draft project and along with my dependencies I might be able to import few packages. However there is the package org.apache.spark.streaming.kinesis that even though it's included in my POM.xml doesnt import it.

My pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.streaming.sparkdemo</groupId>
  <artifactId>spark-demo</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>spark-demo</name>
  <url>http://maven.apache.org</url>

  <dependencies>

  <!-- Import Spark -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>1.4.0</version>
    </dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>

    <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk</artifactId>
    <version>1.11.78</version>  
  </dependency>

  <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.3.0</version>
    <scope>provided</scope>
</dependency>


  <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>amazon-kinesis-client</artifactId>
      <version>1.9.3</version>
</dependency>


  </dependencies>



</project>

However, after build it I get this awful message from my import:

enter image description here

1

There are 1 best solutions below

0
On

I have solved this issue included the ASL Kinesis integration package in my POM.XML

https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl_2.11/2.3.0

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kinesis-asl_2.11</artifactId>
    <version>2.3.0</version>
</dependency>