Can't access Ganglia on EC2 Spark cluster

Question

Launching using spark-ec2 script results in:

Setting up ganglia RSYNC'ing /etc/ganglia to slaves... <...>

Shutting down GANGLIA gmond: [FAILED]

Starting GANGLIA gmond: [ OK ]

Shutting down GANGLIA gmond: [FAILED]

Starting GANGLIA gmond: [ OK ]

Connection to <...> closed. <...> Stopping httpd:
[FAILED] Starting httpd: httpd: Syntax error on line 199 of /etc/httpd/conf/httpd.conf: Cannot load modules/libphp-5.5.so into server: /etc/httpd/modules/libphp-5.5.so: cannot open shared object file: No such file or directory [FAILED] [timing]

ganglia setup: 00h 00m 03s Connection to <...> closed.

Spark standalone cluster started at <...>:8080 Ganglia started at <...>:5080/ganglia

Done!

However, when I netstat, there is no 5080 port listened on.

Is this related to the above error with httpd or it's something else?

EDIT:

So the issue is found (see the answer below), and the fix can be applied locally on the instance, after which Ganglia works fine. However the question is how to fix this issue in the root, so that spark-ec2 script can start Ganglia normally without intervention.

Answer 1

The fact that ganglia is not available is related to these errors - ganglia is php application and it won't run without php module for apache.

Which version of spark you are using to start cluster?

It is wierd error - these file should be present in AMI image.

Answer 2

Just traced the error: /etc/httpd/conf/httpd.conf is trying to load libphp-5.5 library while modules/ contains libphp-5.6 version...

Changing httpd.conf fixes the issue, however I'd be good to know a permanent fix within spark-ec2 script

Answer 3

This is because httpd fails to launch. As you have noted httpd.conf is trying to load modules and failing. You can reproduce the problem via apachectl start and examine exactly what modules are failing to load.

In my case there was one involving "auth" and "core". The last four (maybe five) listed will also fail to load. I did not encounter anything related to PHP so maybe our cases our different. Anyway the hacky solution is to comment out the problems. I did so and am running Ganglia without issue.