Background
We have a production fabric cluster setup and has been been running for a year. Now most of the certs expire and the cluster crash, including both tls and identity certs.
I tried to fix by completely removing old certs and private keys, generate and enroll new identities for peer, peer admin, orderer, orderer admin.
Everything works again, but I cannot instantiate/upgrade chaincode in existing channel because the channel was configured with old admin certs.
Problem
So now look like I'm stuck in a deadlock. In order to update channel config with new cert, I need to sign the update with matching old cert, which is already expired and blocked by orderer.
I find out that we can disable expired cert check in orderer using ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true. But now I don't have the old admin private key so I still cannot update the channel config.
Questions
I already replaced old private keys with new one so there is no way to use the old cert again. Can I do something to resolve this channel issue?
Suggestions are greatly appreciated.
The conclusion is that the orderer and peer's binary can be artificially manipulated and updated.
Now all verify is valid unconditionally. so, update all recent status.
Afterwards, the backed up binary is replanted into the container to solve this problem.