Ambari BigInsights Kafka Not Starting

Question

Well it seems I've hit my first issue with my BigInsights Image, not a massive problem, but something to think about. On my Ambari browser services page it was showing that the Kafka service was not running, I tried a restart a number of times, but this seemed to continuously fail. So I figured that I best look into it a bit further. In this case the issue was on the Ambari Master server which has the most services running on it. So first call of action is to see if maybe Ambari is not making the call correctly:

[root@master ~]# kafka
Usage: /usr/bin/kafka {start|stop|status|clean}
[root@master ~]# kafka status
Kafka is not running.
[root@master ~]# kafka start
Starting Kafka succeeded with PID=15815.
[root@master ~]# kafka status
Kafka is not running.

Next I tired a clean start, not that I figured it would make much difference, but maybe there was a issue with the logs not allowing it to restart:

[root@master ~]# kafka clean
Removed the Kafka PID file: /var/run/kafka/kafka.pid.
Removed the Kafka OUT file: /var/log/kafka/kafka.out.
Removed the Kafka ERR file: /var/log/kafka/kafka.err.
[root@master ~]# kafka status
Kafka is not running. No pid file found.
[root@master ~]# kafka start
Starting Kafka succeeded with PID=15875.
[root@master-01 ~]# kafka status
Kafka is not running.

Answer 1

So lets take a proper look at the logs:

[root@master ~]# ls -ltr /var/log/kafka/
-<cut>-
-rw-r--r-- 1 kafka hadoop  6588 Aug 11 13:55 controller.log.2015-08-11-13
-rw-r--r-- 1 kafka hadoop  6000 Aug 11 13:59 server.log.2015-08-11-13
-rw-r--r-- 1 kafka hadoop  6588 Aug 11 14:55 controller.log
-rw-r--r-- 1 kafka hadoop  5700 Aug 11 14:56 server.log
-rw-r--r-- 1 root  root     284 Aug 11 15:09 kafka.err
-rw-r--r-- 1 root  root     522 Aug 11 15:09 kafka.out
-rw-r--r-- 1 kafka hadoop   707 Aug 11 15:09 kafkaServer-gc.log

Lets look at the error and out files:

[root@master ~]# cat /var/log/kafka/kafka.err
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
[root@master ~]# cat /var/log/kafka/kafka.out
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid15875.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid16305.log

Ah, that's odd, as I asked for at least 4GB of memory for my VMs, lets check:

[root@master ~]# cat /proc/meminfo
MemTotal:        1922260 kB
MemFree:          278404 kB
Buffers:            8600 kB
Cached:            43384 kB

Best get some more memory allocated! Normally the minimum that you should install BigInsights with, as recommended by the IBM support pages is 8GB, so this gives you rather a insight into why. At least 2GB of it is just to run the installed services on the system, even before you start loading the DB and running queries.