GCP Data Fusion Twitter Tweet Stream Error: java.lang.NoClassDefFoundError: org/apache/spark/Logging

Question

I am trying to stream twitter data into a Big Query table using GCP Data Fusion. I've added my twitter credentials to the twitter component and validate with no errors. I also validate the Big Query component with no errors. When I run the preview it stops after around 30 seconds and I get the following error:

java.lang.NoClassDefFoundError: org/apache/spark/Logging

Here is an image of my Data Fusion job

Any help would be greatly appreciated.

12/15/2020: Adding some basic information about my Data Fusion Instance

Logs: https://pastebin.com/PxKpqfCp

Answer 1

Seems to be well known to developers issue, looking through the CDAP issue tracker, I've found the relevant thread PLUGIN-194 describing the same problem in particular affecting Twitter Tweet Stream plugin functionality. Let's track out this case for any further updates occurred.