I am new to Jenkins-x and having trouble with the installation on EKS using Secrets Manager with Terraform installation approach. Seeing these errors. Any help is really appreciated. I did make a dummy commit to our internal bitbucket cluster repo(jx3-eks-asm) and still seeing same errors, I believe it's because the boot pods are failing as well. Here are the errors from health status command.
There were bunch of other issues too but was able to resolve by making local changes to kuberhealthy and other helm chart versions. I have verified that we have terraform, aws-cli, aws-iam-authenticator, and wget as mentioned in the prerequisites section. Noticed that none of the health commands work if kuberhealthy is disabled and it appears that there are bunch of issues with that Helm chart and CRDs installation. Its using an old version of the chart, so I updated to 0.0.90 or similar to fix another issue but now seeing this.
Error from one of boot pods in the jx-git-operator ns:
error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-bot-token-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
error validating "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-kuberhealthycheck.yaml": error validating data: ValidationError(KuberhealthyCheck.spec.podSpec.containers[0]): unknown field "restartPolicy" in io.github.comcast.v1.KuberhealthyCheck.spec.podSpec.containers; if you choose to ignore these errors, turn validation off with --validate=false
make[1]: Leaving directory '/workspace/source'
make[1]: *** [versionStream/src/Makefile.mk:324: kubectl-apply] Error 1
error: failed to regenerate: failed to regenerate phase 1: failed to run 'make regen-phase-1 NEW_CLUSTER=false' command in directory '.', output: ''
make: *** [versionStream/src/Makefile.mk:269: regen-check] Error 1
kubectl version
Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.13-eks-fb459a0", GitCommit:"55bd5d5cb7d32bc35e4e050f536181196fb8c6f7", GitTreeState:"clean", BuildDate:"2022-10-24T20:38:50Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.13-eks-0a21954", GitCommit:"6305d65c340554ad8b4d7a5f21391c9fa34932cb", GitTreeState:"clean", BuildDate:"2023-04-15T00:33:45Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
jx health status -A
NAME NAMESPACE STATUSERROR MESSAGE
daemonset kuberhealthy OK
deployment kuberhealthy OK
dns-status-internal kuberhealthy OK
jx-install jx-git-operator ERRORlatest boot job jx-boot-1997a686-37fb-4704-af81-8505a7518877 has been running for more than 30m0s, it could be stuck
latest boot job jx-boot-1997a686-37fb-4704-af81-8505a7518877 has a failed run
jx-pod-status kuberhealthy ERRORpod: jenkins-x-chartmuseum-6cd76fd747-4hwrc in namespace: jx is in pod status phase Pending
pod: jx-build-controller-b6bdc5b6d-ch2xh in namespace: jx is in pod status phase Pending
pod: jx-pipelines-visualizer-7b685f9c79-bdx5k in namespace: jx is in pod status phase Pending
pod: lighthouse-foghorn-cff4fd4f5-r76kr in namespace: jx is in pod status phase Pending
pod: lighthouse-keeper-57dfddb5f9-plfjn in namespace: jx is in pod status phase Pending
pod: lighthouse-webhooks-85f985b664-9wdgx in namespace: jx is in pod status phase Pending
pod: nexus-nexus-776c96f565-vxp42 in namespace: jx is in pod status phase Pending
jx-secrets kuberhealthy ERRORERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
ERROR, Secrets Manager can't find the specified secret.
jx secret verify
SECRET STATUS
jx-production/tekton-container-registry-auth key tekton-container-registry-auth missing properties: .dockerconfigjson
jx-staging/tekton-container-registry-auth key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/jenkins-maven-settings key jx-maven-settings missing properties: settingsXml, securityXml
jx/jenkins-x-chartmuseum valid: jx-admin-user/BASIC_AUTH_PASS, jx-admin-user/BASIC_AUTH_USER
jx/jx-basic-auth-htpasswd key jx-basic-auth-htpasswd missing properties: token
jx/jx-basic-auth-user-password valid: jx-basic-auth-user/password, jx-basic-auth-user/username
jx/lighthouse-oauth-token key lighthouse-oauth missing properties: token
jx/nexus valid: jx-admin-user/password
jx/tekton-container-registry-auth key tekton-container-registry-auth missing properties: .dockerconfigjson
jx/tekton-git key jx-pipeline-user missing properties: token, username