How to add a new event to Apache Spark Event Log

Question

Apache Spark pushes a bunch of information in event logs. How can I push my custom events into this event log?

Does Spark expose any APIs to do this? Or can I extend any existing class to do this?

Answer 1

Your class can extend org.apache.spark.Logging and use the help methods defined there:

logInfo("task success") 
logWarn("didn't receive data")
...

Note the warning on that trait:

NOTE: DO NOT USE this class outside of Spark. It is intended as an internal utility. This will likely be changed or removed in future releases.

Answer 2

You can extend "StreamingQueryListener" if you have a stream query and then add it to

    spark.streams.addListener(new EventCollector(APP_NAME))

where

 EventCollector(APP_NAME) extends StreamingQueryListener

and write your own methods

onQueryStarted
onQueryProgress
onQueryTerminated

methods and put custom code in it