Google Stackdriver Error Reporting not picking up errors

Question

Logs with severity “ERROR” is not identified by Error and Reporting tool. Application logs are being directed at Google Stackdriver Logging using fluentd agent and some of these are third party java components.

{
 insertId:  "14sf3lvg3ccncgh"   
 jsonPayload: {
   class:  "o.a.w.MarkupContainer"    
   message:  "Unable to find component with id 'search2' in [Form [Component id = form]]
              Expected: 'form:search2'.
              Found with similar names: 'form:search'
        at org.apache.wicket.markup.MarkupStream.throwMarkupException(MarkupStream.java:526) ~[wicket-core-6.22.0.jar:6.22.0]
        at org.apache.wicket.MarkupContainer.renderNext(MarkupContainer.java:1438) ~[wicket-core-6.22.0.jar:6.22.0]
        at org.apache.wicket.MarkupContainer.renderAll(MarkupContainer.java:1557) ~[wicket-core-6.22.0.jar:6.22.0]
        at org.apache.wicket.MarkupContainer.renderComponentTagBody(MarkupContainer.java:1532) ~[wicket-core-6.22.0.jar:6.22.0]
        at org.apache.wicket.MarkupContainer.onComponentTagBody(MarkupContainer.java:1487) ~[wicket-core-6.22.0.jar:6.22.0]"    
   milsec:  "576"    
   reportLocation: {…}    
   serviceContext: {…}    
   tag:  "test.gui"    
   thread:  "[ajp-apr-8009-exec-5]"    
 }
 labels: {…}   
 logName:  "projects/myservice/logs/test.gui"   
 receiveTimestamp:  "2017-08-29T15:20:16.847782870Z"   
 resource: {…}   
 severity:  "ERROR"   
 timestamp:  "2017-08-29T15:20:11Z"   
}

Using the following configuration allows for my application logs to be forwarded correctly to Googles Stackdriver logging and all entries are correctly identified.

<source>
  @type tail
  path /var/log/test/test_gui/test_gui.log
  pos_file /var/lib/google-fluentd/pos/test_gui-multiline.pos
  read_from_head true
  tag test.gui
  format multiline
  time_format %Y-%m-%d %H:%M:%S
  format_firstline /\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3}\s(?<severity>\S*)/
  format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}),(?<milsec>\d{1,3})\s(?<severity>\S*)\s(?<class>\S*)\s(?<thread>\[\S*\])\s(?<message>.*)/
</source>

However for severity ERROR, Error Reporting never noticed these entries.

The output was identified as textPayLoad, I used the following filter, which ensured output was jsonPayload

<filter test.gui>
  @type record_transformer
  <record>
    serviceContext {"service": "test.gui", "version": "1"}
    reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown", "functionName": "unknown"}
   tag ${tag}
  </record>
</filter>

Still the error jsonPayload is being ignored.

If I replace the message using the filter, then suddenly Error Reporting is working

 <filter test.gui>
      @type record_transformer
      <record>
        serviceContext {"service": "test.gui", "version": "1"}
        reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown", 
        "functionName": "unknown"}
        message "java.lang.TestError: msg
           at com.example.TestClass.test (TestClass.java:51)
           at com.example.AnotherClass (AnotherClass.java:25)"
       tag ${tag}
     </record>
    </filter>

How can I force Error Reporting to pick these error entries as my next step would be to implement some form of alerting.

Answer 1

Third parties did not produce correct Java Stack trace. I needed the reportLocation, but this needs to be in context.

I changed the following line:

reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown", "functionName": "unknown"}

to

context { "reportLocation" : {"filePath": "test_gui.log", "lineNumber": 1, "functionName": "unknown"} }

which ensured that the logs are now picked up by Stackdriver Error Reporting.

This is the final version of my filter:

<filter test.gui>
  @type record_transformer
  <record>
    serviceContext {"service": "test.gui", "version": "1"}
    context { "reportLocation" : {"filePath": "test_gui.log", "lineNumber": 1, "functionName": "unknown"} }
   tag ${tag}
  </record>
</filter>