As in the title, mongodb just changed (I think) on it's own member's hosts, excpet arbiter, from for example mongodb-0.mongodb-headless.infrastructure.svc.cluster.local:27017 to IP adresses on the cluster (OKD), which in turn totally blocked whole thing from being repaired
test> rs.config()
{
_id: 'rs0',
version: 8,
term: 24,
members: [
{
_id: 0,
host: '192.168.120.137:30100',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 5,
tags: {},
secondaryDelaySecs: Long("0"),
votes: 1
},
{
_id: 1,
host: 'mongodb-0.mongodb-headless.infrastructure.svc.cluster.local:27017',
arbiterOnly: true,
buildIndexes: true,
hidden: false,
priority: 0,
tags: {},
secondaryDelaySecs: Long("0"),
votes: 1
},
There are 3 members of this mongodb cluster, and every one of them got this change, the worst part is that I cannot change it back, As there is no primary I am unable to use rs.reconfig(), I cannot use rs.stepDown(), I am unable to use rs.initiate as every one of this pods thinks it's still part of rs. I am unable to do anything from arbiter as it calls for privilges with error:
> MongoServerError: not authorized on admin to execute command {
> replSetGetConfig: 1, $db: "admin" }
It looks like Mongodb just choose to break itself, as I did not do any commands nor anyone else. All I can do is to restart pods, or change their configuration but nothing helps as even If I reset them all this configuration persists.
Did anyone happend to have problem like this?