Debugging Apache Spark clustered application from Eclipse

Question

I am trying to debug Spark application running on eclipse in clustered/distributed environment but not able to succeed. Application is java based and I am running it through Eclipse. Configurations to spark for Master/worker is provided through Java only.

Though I can debug the code on driver side but as the code flow moves in Spark(i.e call to .map(..)), the debugger doesn't stop. Because that code is running in Workers JVM.

Is there anyway I can achieve this ?

I have tried giving following configurations in Tomcat through Eclipse : -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=7761,suspend=n

and setting respective port in Debug->remote java application.

But after these settings I get the error: Failed to connect to remote VM. Connection Refused

If anybody has any solution to this, please help.

Answer 1

I was facing the same issue while configuring the spark debugging on remote master. But after that I've installed the spark on my Ubuntu machine then it worked fine. If you really want to debug, my suggestions are

1- configure spark on your testing machine then you can easily debug applications.
2- use IntelliJ IDEA, I've used it for for debugging if I've to use remote spark.

EDITED: If you are going to use IntelliJ IDEA then you can easily configure remote debugging as explained here. Debugging Apache Spark Jobs