I am using TKGI to deploy my applications on Clusters. Suddenly we lost access to our Clusters, We are successfully able to login to TKGI but when we try to get credentials for a specific cluster, by executing the following command.
pks get-credentials abc-cluster
It hangs for a certain time and gives the below error.
Error: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: f617fec2-2748-48cb-98c8-17dc4e63f666, broker-request-id: a15f4a9b-4b1f-4bfe-a4d6-7f58f77a3359, operation: bind - error-message: gathering deployment list Cannot get the list of deployments: Finding deployments: Performing request GET 'https://x.x.x.x:25555/deployments?exclude_configs=true': Performing GET request: Requesting token via client credentials grant: Performing request POST 'https://x.x.x.x:8443/oauth/token': Performing POST request: Retry: Post https://x.x.x.x:8443/oauth/token: dial tcp x.x.x.x:8443: connect: connection refused
so we tried executing bosh commands to see what was the issue, That is when we realized we are not able to connect to Bosh Director from our Ops Manager and unfortunately we have lost our pass phrase and have no access to Ops Manager Console. We can only SSH in to Bosh VM with relevant pem file.
We can also see that there is no service running in Bosh VM at port 25555 as expected by TKGI. We get below response when we execute
sudo lsof -i -P -n
not sure where to go from here, Any pointers ?