I have set up an Apache Storm cluster locally and submitted a topology as well which is visible in the Storm UI. Once everything is up and running I am getting the following exception in ui.log file. The error is somewhat related to storm-kafka-monitor which is a bash script and I am running this on windows.
Any help will be highly appreciated. Thanks in advance :)
2018-06-05 11:26:20.293 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception thrown while getting lag for spout id: kafka-spout and spout class: org.apache.storm.kafka.KafkaSpout
2018-06-05 11:26:20.294 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception message:Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
java.io.IOException: Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_162]
at org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:180) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.TopologySpoutLag.getLagResultForOldKafkaSpout(TopologySpoutLag.java:193) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:59) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.ui.core$fn__9586.invoke(core.clj:1165) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.compojure.core$make_route$fn__5979.invoke(core.clj:100) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.compojure.core$if_route$fn__5967.invoke(core.clj:46) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.compojure.core$if_method$fn__5960.invoke(core.clj:31) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.compojure.core$routing$fn__5985.invoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
at org.apache.storm.shade.compojure.core$routes$fn__5989.invoke(core.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8894.invoke(cors.clj:149) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8841.invoke(json.clj:56) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7904.invoke(reload.clj:22) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.ui.helpers$requests_middleware$fn__6874.invoke(helpers.clj:52) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.ui.core$catch_errors$fn__9761.invoke(core.clj:1428) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__6541.invoke(keyword_params.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__6584.invoke(nested_params.clj:84) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__6513.invoke(params.clj:64) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__6836.invoke(flash.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__6822.invoke(session.clj:98) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__6371.invoke(servlet.clj:127) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.util.servlet$servlet$fn__6375.invoke(servlet.clj:136) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6967.invoke(helpers.clj:192) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) ~[storm-core-1.2.1.jar:1.2.1]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_162]
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) ~[?:1.8.0_162]
at java.lang.ProcessImpl.start(ProcessImpl.java:137) ~[?:1.8.0_162]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_162]
... 68 more
As you note the script is a bash file. The feature (showing how far behind the log end offset the spout is in Storm UI) doesn't work on Windows currently. It's a little inconvenient, but it shouldn't prevent your topologies from working correctly.
If you want to know how far behind the log end offset the spout is, you can use the kafka-consumer-groups.bat script on one of your Kafka machines instead. If you're using the
storm-kafka-clientspout, the spout lag is also part of the metrics provided by that component.I'm hoping we can get rid of
storm-kafka-monitorat some point (it has some other issues https://issues.apache.org/jira/browse/STORM-2600), but until then, if you'd like to try providing Windows compatibility, you'd need to port https://github.com/apache/storm/blob/master/bin/storm-kafka-monitor to cmd/powershell, and modify https://github.com/apache/storm/blob/4137328b75c06771f84414c3c2113e2d1c757c08/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L146 so it calls either the Windows or bash script depending on OS.