How to specify time range in cruise control for Kafka?

Question

Trying to access API which is given cruise control for Apache Kafka. for a get call, it says

• Get the partition load sorted by the utilization of a given resource and filtered by given topic regular expression and partition number/range
  GET /kafkacruisecontrol/partition_load?resource=[RESOURCE]&start=[START_TIMESTAMP]&end=[END_TIMESTAMP] &topic=[topic]&partition=[partition/start_partition-end_partition]

and

• NOTE: All the timestamps are epoch time in second granularity (given in the doc)

So here's my constructed URL (as per my requirement I just need start and end time)

http://localhost:8081/kafkacruisecontrol/partition_load?start=1533101050920&end=1533102938311&json=true

and when I hit this it throws following exception

{
"errorMessage": "Error processing GET request '/partition_load' due to 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range [1533101050920, 1533102938311]'.",
"stackTrace": "java.util.concurrent.ExecutionException: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range [1533101050920, 1533102938311]\n\tat java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)\n\tat java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)\n\tat com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.getAndMaybeReturnProgress(KafkaCruiseControlServlet.java:1347)\n\tat com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.getPartitionLoad(KafkaCruiseControlServlet.java:716)\n\tat com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGet(KafkaCruiseControlServlet.java:361)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:687)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:564)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\tat org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)\n\tat org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range [1533101050920, 1533102938311]\n\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.clusterModel(KafkaCruiseControl.java:352)\n\tat com.linkedin.kafka.cruisecontrol.async.GetClusterModelInRangeRunnable.getResult(GetClusterModelInRangeRunnable.java:36)\n\tat com.linkedin.kafka.cruisecontrol.async.GetClusterModelInRangeRunnable.getResult(GetClusterModelInRangeRunnable.java:15)\n\tat com.linkedin.kafka.cruisecontrol.async.OperationRunnable.run(OperationRunnable.java:45)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\t... 1 more\nCaused by: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range [1533101050920, 1533102938311]\n\tat com.linkedin.cruisecontrol.monitor.sampling.aggregator.MetricSampleAggregator.aggregate(MetricSampleAggregator.java:197)\n\tat com.linkedin.kafka.cruisecontrol.monitor.sampling.aggregator.KafkaPartitionMetricSampleAggregator.aggregate(KafkaPartitionMetricSampleAggregator.java:150)\n\tat com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.clusterModel(LoadMonitor.java:423)\n\tat com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.clusterModel(KafkaCruiseControl.java:346)\n\t... 8 more\n",
"version": 1  }

am I missing something?

for reference and here is the git hub link for cruise control API

Answer 1

The exception you're getting seems to be this.

This seems to be triggered when there are not enough valid windows in the time period you've specified. As the time period you've specified is about 30 minutes, is it possible that it may not a large enough time period to make this request.

According to the Cruise Control readme:

The metrics of a newly up broker may take a few minutes to get stable. Cruise Control will drop the inconsistent metrics (e.g when topic bytes-in is higher than broker bytes-in), so first few snapshot windows may not have enough valid partitions.

This could also be related. How long have you been running the cluster before making this request?

Answer 2

I had the similar issue and turns out that the __CruiseControlMetrics topic message version is too low.

message.format.version > 0.9.0

cruise control needs newer version of message so that it can get the message header data. if your message format version set too low, message parsing will fail and no training data will be generated.