I am trying to start an Apache Ignite Docker container (2.15.0 or 2.15.0-jdk11). We need to get the metrics from the container with OpenCensus.
Now the documentation says there is an OpenCensus exporter: https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics-system
And here we have the corresponding class: https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/metric/opencensus/OpenCensusMetricExporterSpi.html
I did what I read in the documentation and added a few things to the default-config.xml which i can successfully use when starting my docker container with this docker-compose.yml:
services:
ignite:
image: apacheignite/ignite:2.15.0-jdk11
environment:
- IGNITE_QUIET=false
- IGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
- IGNITE_NO_ASCII=true
- JAVA_OPTS="Djava.net.preferIPv4Stack=true"
volumes:
- ./config:/opt/ignite/apache-ignite/config/
ports:
- 11211:11211
- 47100-47110:47100-47110
- 47500-47509:47500-47509
- 49112:49112
- 10800:10800
This is in the configuration file (default-config.xml):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<property name="metricsLogFrequency" value="5000"/>
<property name="metricExporterSpi">
<list>
<bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi"/>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_NODE_SEGMENTED"/>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache"/>
<!-- Enable statistics for the cache. -->
<property name="statisticsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</beans>
So the problem now is: Basically the metrics/monitoring works, as long as I don't have that line in the default-config.xml:
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi"/>
Then I get this error from spring:
WARNING: Unknown module: jdk.internal.jvmstat specified to --add-exports
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/opt/ignite/apache-ignite/libs/ignite-core-2.15.0.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Ignite Command Line Startup, ver. 2.15.0#20230425-sha1:f98f7f35
2023 Copyright(C) Apache Software Foundation
class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1150)
at org.apache.ignite.Ignition.start(Ignition.java:328)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:365)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:381)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)
at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:698)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:883)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:808)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:647)
at org.apache.ignite.Ignition.start(Ignition.java:325)
... 1 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ignite.cfg' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]: Cannot create inner bean 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' of type [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] while setting bean property 'metricExporterSpi' with key [2]; nested exception is org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] for bean with name 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:389)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:127)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:428)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1702)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1447)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551)
at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:375)
... 9 more
Caused by: org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi] for bean with name 'org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi#71e9ddb4' defined in URL [file:/opt/ignite/apache-ignite/config/default-config.xml]; nested exception is java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1486)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:488)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:374)
... 24 more
Caused by: java.lang.ClassNotFoundException: org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:284)
at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:469)
at org.springframework.beans.factory.support.AbstractBeanFactory.doResolveBeanClass(AbstractBeanFactory.java:1551)
at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1478)
... 26 more
Failed to start grid: Failed to instantiate Spring XML application context (make sure all classes used in Spring configuration are present at CLASSPATH) [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml]
Note! You may use 'USER_LIBS' environment variable to specify your classpath.
So finally my question: Am I missing a step? Do I have to manually provide the SPI class for the container? Or is there something else that I can do?
I have tried to adjust the configuration file in many ways, but this seems to be the correct structure.
Update:
I have adjusted the docker-compose.yml, so now it looks like this:
services:
ignite:
image: apacheignite/ignite:2.15.0-jdk11
environment:
- IGNITE_QUIET=false
- IGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
- IGNITE_NO_ASCII=true
- OPTION_LIBS=ignite-opencensus,ignite-web
- JAVA_OPTS="Djava.net.preferIPv4Stack=true"
#- EXTERNAL_LIBS = "C:/git/ignite-prometheus/target/ignite-prometheus-1.0.0-jar-with-dependencies.jar"
volumes:
- ./config:/opt/ignite/apache-ignite/config/
- ./libs:/opt/ignite/apache-ignite/libs/user_libs
ports:
- 11211:11211
- 47100-47110:47100-47110
- 47500-47509:47500-47509
- 49112:49112
- 10800:10800
- 9000:9000
(Note: I'm testing locally, so I have Windows paths. This will ofc not be the final version.)
Additionally I have provided a Jar file with the needed dependencies for my new beans in the configuration. (Taken from the blog post mentioned in the answer by Stephen).
So now my default-config.xml looks like this: (I excluded the beans tag for this.)
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<property name="metricsLogFrequency" value="60000"/>
<property name="metricExporterSpi">
<list>
<bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
<bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi">
<property name="period" value="60000" />
</bean>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_NODE_SEGMENTED"/>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache"/>
<property name="statisticsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
<bean id="opencensusWrapper" class="org.springframework.beans.factory.config.MethodInvokingBean">
<property name="staticMethod" value="io.opencensus.exporter.stats.prometheus.PrometheusStatsCollector.createAndRegister"/>
</bean>
<bean id="httpServer" class="io.prometheus.client.exporter.HTTPServer">
<constructor-arg type="java.lang.String" value="localhost"/>
<constructor-arg type="int" value="9000"/>
<constructor-arg type="boolean" value="true"/>
</bean>
The Ignites spring framework does something with it. Because without the Jar file I would get corresponding ClassNotFoundExceptions.
Now maybe the structure of the configuration could be messed up, but I don't have any idea what I could do to reach my goal.
Note that in the logs of the Ignite container I don't see any sign of a web server being started. And no sign of port 9000 being used.
But I also don't get any errors in the container log.
OpenCensus support is an optional module. You need an add another environment variable.
You'll also need some way to export the metrics. I wrote a blog about this, though it isn't specific to Docker.