This question is NOT a generic question about how to correctly set a Service Account with the proper Role(s) in order to access K8s resources. It is about to ask for help in the context of a failure at Spring application start up due to a problem with accessing a config map after an upgrade of Spring Boot and Spring Cloud, with something that was working perfectly before.
So the question is: what change in configuration, or in any dependency, has the upgrade brought, that we are not aware of, that is breaking the access to the config maps at start up by the fabric8 KubernetesClient?? I am still investigating if this is the same issue reported in How to make configuration with discovery first lookup work with spring cloud 2022.0.3
The starting point was a micro developed on top of Spring Boot 2.7.14 and Spring Cloud 2021.0.8. With K8s 1.25 the application was starting without any problem and config maps could be read at start up to get configuration to set up, among other things, jetty parameters.
After the upgrade to Spring Boot 3.1.5 and Spring Cloud 2022.0.3 something got broken and I got the problem below:
2023-11-15T15:28:20.472Z WARN 1 --- [ main] o.s.c.k.c.c.ConfigUtils : Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/fesvc-namespace/configmaps. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:fesvc-namespace:fesvc-sa" cannot list resource "configmaps" in API group "" in the namespace "fesvc-namespace".. Ignoring.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/fesvc-namespace/configmaps. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:fesvc-namespace:fesvc-sa" cannot list resource "configmaps" in API group "" in the namespace "fesvc-namespace".
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) ~[kubernetes-client-api-6.2.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:517) ~[kubernetes-client-6.2.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:404) ~[kubernetes-client-6.2.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:378) ~[kubernetes-client-6.2.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:88) ~[kubernetes-client-6.2.0.jar:?]
at org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigMapsCache.lambda$byNamespace$0(Fabric8ConfigMapsCache.java:56) ~[spring-cloud-kubernetes-fabric8-config-3.0.3.jar:3.0.3]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
...
Next is the org.springframework.context.ApplicationContextException since the web server start fails because properties needed to create the bean of type WebServerFactoryCustomizer cannot be read. So the problem seems to be, unexpectedly, in the access to the config maps during boot time, something that was not failing before. And by this I mean that nothing was changed in the K8s specs of the application: we expected that everything worked as before.
Relevant parts of the poms:
parent pom:
<dependencyManagement>
<dependencies>
<!-- Spring Boot dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring.boot.dependencies}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring.cloud.dep.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
...
</dependencies>
</dependencyManagement>
intermediate pom:
<dependencies>
...
<!-- Spring Framework dependencies start -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-fabric8-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty.http2</groupId>
<artifactId>http2-server</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-jmx</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ws</groupId>
<artifactId>spring-ws-core</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-sftp</artifactId>
</dependency>
<!-- Spring Framework dependencies end -->
...
</dependencies>
K8s version:
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.25.2
Nevertheless, I've checked that everything was properly set up from the k8s side, and I think it is: there is a role to define the set of permissions, a service account to access the resources from the API and a rb attaching both, along with the projected volume in the pod where the token is supposed to be stored:
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
ericsson.com/product-name: fesvc
ericsson.com/product-number:
ericsson.com/product-revision:
meta.helm.sh/release-name: fesvc-tests
meta.helm.sh/release-namespace: fesvc-namespace
creationTimestamp: "2023-11-14T14:42:50Z"
labels:
app.kubernetes.io/instance: fesvc-tests
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: fesvc
app.kubernetes.io/version:
chart: fesvc-1.17.0-3
name: fesvc-sa
namespace: fesvc-namespace
resourceVersion: "690"
uid: 010b3cb2-0ff2-4059-ba14-3b6b04fa3d70
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
ericsson.com/product-name: fesvc
ericsson.com/product-number:
ericsson.com/product-revision:
meta.helm.sh/release-name: fesvc-tests
meta.helm.sh/release-namespace: fesvc-namespace
creationTimestamp: "2023-11-14T14:42:50Z"
labels:
app.kubernetes.io/instance: fesvc-tests
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: fesvc
app.kubernetes.io/version:
chart: fesvc-1.17.0-3
name: fesvc-api-role
namespace: fesvc-namespace
resourceVersion: "714"
uid: f0b76567-b854-4bcf-a423-042059cd51c3
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
ericsson.com/product-name: fesvc
ericsson.com/product-number:
ericsson.com/product-revision:
meta.helm.sh/release-name: fesvc-tests
meta.helm.sh/release-namespace: fesvc-namespace
creationTimestamp: "2023-11-14T14:42:50Z"
labels:
app.kubernetes.io/instance: fesvc-tests
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: fesvc
app.kubernetes.io/version:
chart: fesvc-1.17.0-3
name: fesvc-api-rolebinding
namespace: fesvc-namespace
resourceVersion: "721"
uid: 0cacd75b-7d30-4e4e-a4d5-f32a312dacba
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: fesvc-api-role
subjects:
- kind: ServiceAccount
name: fesvc-sa
namespace: fesvc-namespace
I am answering myself just in case it is of help for someone.
Something has changed in the combination io.fabric8 / Spring Cloud that I have not found in any documentation, but the fact is that after upgrading from Spring Cloud 2021.0.8 to 2022.0.3 something seems to have changed in the RBAC handling with Service Accounts, with the same version of Kubernetes.
The error I reported gets fixed adding the "list" verb to the Role we used to get cm's with the service account, like so:
With this changed (I insist, not required before) everything works as it did before the upgrade.