How to analyze over log files with custom logback pattern configuration

Question

I would like to analyze through set of log files (look out for errors and creating a report)

These log files has the records in custom logback pattern

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
        <Pattern>%d{"yyyy-MM-dd'T'HH:mm:ss,SSSXXX", UTC} [%thread] %-5level %logger{36} - %msg%n</Pattern>
    </layout>
</appender>

and the sample log entry

2018-02-05 07:59:49,140 22425254620 [http-11.11.11.11:8000-7] ERROR com.sample - Error in processing

I a using windows machine.. I looked through logalyze and apache-chainsaw but unable to parse the log file properly (error records are not displayed properly) Am i doing anything wrong in using the above?

Instead of doing a manual check, Is there any way/open-source tool to look through the entire set of log files and fetch the error records?

Answer 1

It seems that the Pattern from your config file doesn't match the sample log entry: the date format

yyyy-MM-dd'T'HH:mm:ss,SSSXXX

should produce an output like

2018-02-10T14:37:48,187Z

Note the differences with the sample log entry you pasted here: 'T' character between the time and the date, and the end ",187Z" meaning 187 ms Zulu time (XXX means timezone, I just tested this pattern with Logback 1.2.3). But you say that your sample log entry looks more like

2018-02-05 07:59:49,140 22425254620

So maybe you pasted on this page the Pattern you're using for STDOUT, but you're using another one for your log file?

Anyways, to analyze your log file, I would suggest using LogMX. Using the free version, go to "File" > "Configure Log Parsers...", click on the "+" green button at the right to create a Parser, then click on the tab "Log4j/Logback pattern" tab, then paste one of these Patterns: (not sure the one you're using right now -- make sure to copy/paste from my answer, whitespaces are important)

• For logs like 2018-02-05T07:59:49,140Z [http-11.11.11.11:8000-7] , use:

%d{"yyyy-MM-dd'T'HH:mm:ss,SSSXXX",UTC} [%thread] %-5level %logger{36} - %msg%n

• For logs like 2018-02-05 07:59:49,140 22425254620 [http-11.11.11.11:8000-7] , use:

%d{"yyyy-MM-dd HH:mm:ss,SSS"} %mx{?} [%thread] %-5level %logger{36} - %msg%n

You said you have multiple log files: LogMX can open multiple log files and display their content in a single merged view:

• if all the files are in the same directory (local or remote), simply go to "File" > "Open logs...", then check the tab "Merge" at the bottom

• if all the files are NOT in the same directory, go to "File" > "Open several logs in a merged view", then click the "+" green button at the right for each file you want to include

• if all the files are already loaded in LogMX, you can use "Tools" > "Merge logs..."

In order to show only ERRORs in those files, once they are loaded, just use the slider at the bottom left corner of LogMX window to select ERROR (red end)