Knowage stops responding [on Apache Tomcat]

Question

I'm using Knowage and have setup a data source, many parameterized datasets, and a cockpit.

I have a detailed report that requires many datasets.
The cockpit started freezing and I have to restart the Knowage server to get it to respond.

I split the report into different pages, and it seemed to help because it only loads some of the data at a time.
But now I added another dataset [14 total] and the server crashed again.

I'm monitoring the server through a jmx agent to understand why it's hanging, but the utilization seems fine.
Tomcat just hangs and never responds.

It appears the servlet has stopped processing requests, but I don't see any errors in the logs that look fatal.
The heap utilization is not bad, but does look anomalous.

Top output -

Errors in log -

[http-nio-8080-exec-17] 09 Mar 2021 15:46:49,390 ERROR it.eng.spagobi.api.v3.DataSetMainDTO.isGeoDataSet:51 - Error during check of Geo spatial column
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (String)"<?xml version="1.0" encoding="UTF-8"?>
<META version="1">
  <COLUMNLIST>
    <COLUMN alias="label" fieldType="ATTRIBUTE" multivalue="false" name="label" type="java.lang.String"/>
    <COLUMN alias="amount" fieldType="MEASURE" multivalue="false" name="amount" type="java.math.BigDecimal"/>
  </COLUMNLIST>
  <DATASET>
    <PROPERTY name="resultNumber" value="2"/>
  </DATASET>
</META>
"; line: 1, column: 2]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1892)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:747)
    at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4129)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
    at it.eng.spagobi.utilities.json.JSONUtils.toJSONObject(JSONUtils.java:98)
    at it.eng.spagobi.analiticalmodel.execution.service.ExecuteAdHocUtility.hasGeoHierarchy(ExecuteAdHocUtility.java:128)
.......
[http-nio-8080-exec-9] 09 Mar 2021 15:50:48,509 ERROR it.eng.spagobi.rest.interceptors.RestExceptionMapper.toResponse:59 - Catched service error:
javax.ws.rs.NotSupportedException: RESTEASY003065: Cannot consume content type
    at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:403)
    at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:137)
    at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:43)
    at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:48)
    at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:445)
    at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:260)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:197)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at sun.reflect.GeneratedMethodAccessor973.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
    at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
    at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
    at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at sun.reflect.GeneratedMethodAccessor977.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
    at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
    at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
    at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
    at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
    at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
    at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
....

Answer 1

I believe the issue was caused by the limit of DB connections, and max wait in server.xml.

I noticed an error about attributes in the jndi configuration, and updated them.

The freezing has not occurred again, yet.

<!-- KNOWAGE -->
        <Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxTotal="20" maxIdle="10"
            maxWaitMillis="-1" name="jdbc/knowage" type="javax.sql.DataSource"
            url="jdbc:mysql://localhost:3306/knowagedb" username="knowageuser" password="knowagepassword" />
        
        <!-- KNOWAGE CACHE -->
        <Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxTotal="20" maxIdle="4"
            maxWaitMillis="30000" name="jdbc/ds_cache" type="javax.sql.DataSource"
             url="jdbc:mariadb://localhost:3310/cache" username="root" password="root" />

I've made these changes, along with jmx export to prometheus, and a few memory tweaks to make knowage run better on kubernetes.

Have a look here -
https://github.com/savantly-net/knowage-docker