Dump Kinesis Client Library (KCL) logs to file

Question

Am using Kinesis Client Library (KCL) to subscribe to kinesis stream. All the KCL logs are printed on console. I need to dump all the logs to file. I tried adding log4j.properties and common-logging.properties files in src folder, but not able to sort out.

Answer 1

You could also use logback.xml

./bin/kcl-bootstrap --java /usr/local/openjdk-8/bin/java --log-configuration ./properties/logback.xml -p ./properties/kcl.properties

Create logback.xml file with following contents

<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
<appender name="DEBUG" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d [%thread] %-5level %logger{36} [%mdc{ShardId:-NONE}] - %msg %n</pattern>
    </encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>smlogs.log</file>
    <append>true</append>
    <!-- set immediateFlush to false for much higher logging throughput -->
    <immediateFlush>true</immediateFlush>
    <!-- encoders are assigned the type
        ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
    <encoder>
    <pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
    </encoder>
</appender>
<root level="DEBUG">
    <appender-ref ref="DEBUG" />
      <appender-ref ref="FILE" />
</root>

Answer 2

KCL uses Apache Commons Logging (common-logging) in the background.

Try using slf4j with jcl-over-slf4j dependency included (or logback).

That way, your general log collector (slf4j) will handle both your application logs and the logs generated by org.apache.commons.logging.Log interface in Amazon classes.

• Sample class in KCL: https://github.com/awslabs/amazon-kinesis-client/blob/c6e393c13ec348f77b8b08082ba56823776ee48a/src/main/java/com/amazonaws/services/kinesis/metrics/impl/CWPublisherRunnable.java
• SLF4J Bridging: http://www.slf4j.org/legacy.html