Riak - partitions waiting to handoff

Question

I'm running a cluster with 24 nodes with 1024 partitions riak_kv_version : <<"2.1.7-226">>

I have 142 partitions waiting to handoff for more than 10 days. There's no ongoing transfer in the cluster. Under this node [email protected], I can see this error message

<0.30120.441>@riak_core_handoff_sender:start_fold:282 hinted transfer of riak_kv_vnode from '[email protected]' 994791641816054037097625320706298110058774396928 to '[email protected]' 994791641816054037097625320706298110058774396928 failed because of error:{badmatch,{error,closed}} [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,132}]}] <0.9143.441>@riak_core_handoff_sender:start_fold:282 hinted transfer of riak_kv_vnode from '[email protected]' 616571003248974668617179538802181898917346541568 to '[email protected]' 616571003248974668617179538802181898917346541568 failed because of error:{badmatch,{error,closed}} [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,132}]}]

When I check the partitions list (riak-admin cluster partitions) I notice that all partitions which are waiting for handoff are marked as secondary. I was expecting all those partitions type to be primary

Any idea about how to fix this issue?

I have remove for example a node where there are partitions waiting for handoff from the ring, and put it back to the ring. Still seeing the same issue

Answer 1

What is the output of riak-admin ring-status? If ring-ready is false then something is wrong e.g. a node is part way through leaving or joining. Check riak-admin cluster status to find this out. If any nodes in there show !down then mark them down with riak-admin down <nodename> and riak-admin ring-status should then switch ring-ready to true. You may also wish to double check that transfers are allowed i.e. your transfer limit is not set to 0 by running riak-admin transfer-limit. If it is set to 0, you can update it with riak-admin transfer-limit 2 to return it to the default setting. The maximum possible value is 8.

The Riak KV installable package is a collection of components with different versions. The package Riak KV 2.1.7 does not exist, but there is an internal component called riak_kv with version <<"2.1.7-226">>. To help better, I need to know the installed version of Riak KV (which then tells us the versions of all components) and not just the version of one component. Please determine your Riak KV version by running riak version for KV 2.x and earlier or riak versions for KV 3.x and higher.

For people using Riak KV 3.x who refer to this question later, please substitute riak-admin in any commands for riak admin.