Cloud Automation Manager Pods on CrashLoopBackOff

Question

I'm having an issue where some of my Pods are on CrashLoopBackOff when I try to deploy CAM from the Catalog. I also followed the instructions in the IBM documentation to clear the data from PVs (By doing rm -Rf /export/CAM_db/*) and purge the previous installations of CAM.

Here are the pods that are on CrashLoopBackOff: Cam Pods

Here's the specific error when I describe the pod: MongoDB Pod

Answer 1

Ro- It is almost always the case that if the cam-mongo pod does not come up properly, the issue is with the PV unable to mount/read/access the actual disk location or the data itself which is on the PV.

1. Since your pod events indicates container image already exists, and scoped to the store, it seems like you have already tried before to install CAM and its using CE version from the Docker store, correct?
2. If a prior deploy did not go well, do clean up the disk locations as per the doc, https://www.ibm.com/support/knowledgecenter/SS2L37_3.1.0.0/cam_uninstalling.html but like you showed I can see you already tried by cleaning CAM_db, so do the same for the CAM_logs, CAM_bpd and CAM_terraform locations.
3. Make a note of our install troubleshooting section as it describes a few scenarios in which CAM mongo can be impacted: https://www.ibm.com/support/knowledgecenter/SS2L37_3.1.0.0/ts_cam_install.html
4. in the bottom of the PV Create topic, we provide some guidance around the NFS mount options that work best, please review it: https://www.ibm.com/support/knowledgecenter/SS2L37_3.1.0.0/cam_create_pv.html

Hope this helps you make some forward progress!

Answer 2

The postStart error you can effectively ignore, it means mongo container probably failed to start, so it kills a post script.

This issue usually is due to NFS configuration issue.

I would recommend you to try the troubleshooting steps here in the section that has cam-mongo pod is in CrashLoopBackoff

https://www.ibm.com/support/knowledgecenter/SS2L37_3.1.0.0/ts_cam_install.html

If it's NFS, typically it's things like

-no_root_squash is missing on base directory

-fsid=0 needs to be removed on the base directory for that setup

-folder permissions.

Note. I have seen another customer experiencing this issue and the problem was caused by NFS: there were .snapshot file there already, they have to remove it at first.