I have elasticsearch, kibana, apm-server setup in a ec2 instance. APM server is setup and getting data from other application server instances.
When I had a look into stack management apm-7.6.0 related indices have errors.
ilm.step:ERROR
apm-7.6.0-error-000001
apm-7.6.0-span-000001
apm-7.6.0-profile-000001
apm-7.6.0-transaction-000001
apm-7.6.0-metric-000001
_GET /apm-7.6.0-span-000001/_ilm/explain -> query
"step_info" : {
"type" : "security_exception",
"reason" : "action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]",
"stack_trace" : """ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]]
Error shows that I am using kibana user for apm-server which dont have ilm access,but I am using a separate user 'apm-server-kibana' with kibana_system,kibana_admin,apm_system,apm-ilm roles..I have added 'all'access for ilm for apm* indices using apm-ilm role .
ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-error-000001],
this action is granted by the index privileges [manage,all]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:35)
at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:656)
at org.elasticsearch.xpack.security.authz.AuthorizationService.access$300(AuthorizationService.java:101)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:704)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:689)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:659)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:556)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:336)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:599)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:290)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:289)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$5(RBACEngine.java:328)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:352)
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:325)
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:300)
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:265)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:229)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:127)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:161)
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:278)
at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:133)
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:121)
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:231)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:330)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:391)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:402)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:327)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:268)
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:161)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77)
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:196)
at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:52)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1286)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.updateSettings(AbstractClient.java:1672)
at org.elasticsearch.xpack.core.ilm.UpdateSettingsStep.performAction(UpdateSettingsStep.java:42)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:290)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.clusterStateProcessed(IndexLifecycleRunner.java:246)
at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:523)
at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:410)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:410)
at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:270)
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:262)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:239)
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
In Kibana.yml
elasticsearch.username: kibana
In apm-server.yml
I am not using user 'kibana' anywhere but using 'apm-server-kibana'
Why this error shows as Kibana user?
How to fix this error?
This apm rollover policies are created by default when using apm and these policies uses the default user 'kibana' to create it.. So Kibana user dont have access for update.
So as per documentation line if I modify the default apm rollover policy with the logged in user[having access for update ilm],then select the 'retry index' option has solved this error.
Documentation: If you use Elasticsearch’s security features, ILM performs operations as the user who last updated the policy. ILM only has the roles assigned to the user at the time of the last policy update.