I am trying to write unit test case for my spark bigquery implementation. Say, If I want to test this below piece of code:
val wordsDF = (spark.read.format("bigquery")
.option("table","bigquery-public-data:samples.shakespeare")
.load()
.cache())
And do some assertion on wordsDF
I am trying to think on the lines of:
- Embedded kafka for kafka
- Embedded cassandra for cassandra
Do we have any such library or alternative that can be used to write spark bigquery unit test