Riak failed to start within 15 seconds after renaming node

Question

sudo riak start does not work anymore after I change nodename from [email protected] to [email protected]. I followed the steps mentioned here: https://docs.riak.com/riak/kv/latest/using/cluster-operations/changing-cluster-info/index.html. I am using Oracle VM, Ubuntu 18.04 , installed Riak 2.2.5.

The error is: Riak failed to start within 15 seconds, see the output of 'riak console' for more information. If you want to wait longer, set the environment variable WAIT_FOR_ERLANG to the number of seconds to wait.

Riak console gives me a part not okay: Root: /usr/lib/riak Erlang R16B02_basho10 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [hipe] [kernel-poll:true] [frame-pointer]

[os_mon] memory supervisor port (memsup): Erlang has closed [os_mon] cpu supervisor port (cpu_sup): Erlang has closed

Does someone have similar issues?

Answer 1

That sounds like some kind of config related issue. Did you follow all the re-ip steps to the letter? Is there any data on this node you need to keep? If not, probably the easiest way to fix this is to delete the ring directory or failing that, the entire data directory.

That said, before deleting stuff, you might want to check whether any Riak or beam processes (ps aux | grep [b]eam and ps aux | grep [r]iak) have been left running and kill those processes before trying to start again.

You may also wish to look at /var/log/riak/console.log and share the contents of the entire last start up attempt in this question and future ones. Generally, the more relevant information you share, the easier it is for the community to help you.

Erlang has closed is usually sent after it told you what went wrong either in riak console or in the log files. Having all the things it told you before that point would also be beneficial to aid with diagnosis.

Speaking as the package manager for Riak, did you build this version of Riak yourself? Ubuntu 18 never had a 2.2.5 release. See https://files.tiot.jp/riak/kv/2.2/2.2.5/ubuntu/ and you'll notice there is no package for Bionic. Version 2.2.5 had a tagging issue where two of different versions of dependencies had conflicting tags and when you built it, it could result in problems in the final build. To avoid this, 2.2.6 is the official first open source release and does have a Ubuntu 18.04 (Bionic) release at https://files.tiot.jp/riak/kv/2.2/2.2.6/ubuntu/ which you can find in the appropriate subfolder.

Given the potential risk of using 2.2.5 in production based on which versions of the dependencies that git chose to pull in at time of build, you may wish to consider taking this opportunity to update to 2.2.6 or 2.9.10 (https://files.tiot.jp/riak/kv/2.9/2.9.10/ubuntu/) as both can do in-situ upgrades from a 2.2.5 setup. The upgrade process would be to stop the node, optionally back up the data directory, install the new package, make any necessary changes to the config files and start the node again. The full process is documented at https://www.tiot.jp/riak-docs/riak/kv/2.9.0p5/setup/upgrading/version/ but if you are already running 2.2.5, many of the steps listed will not apply.