Apache access log parsing on fluentd

1.4k Views Asked by At

Can someone help me for Apache access log parsing on fluentd?

We are using Kibana to view the logs, but we can't able to split the message part using parser. below is our sample access log,

Sample Apache access log

127.0.0.1 - - [07/Sep/2017:04:52:19 +0000] "GET /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" 200 32775 5113 "domain.test.com" "http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"

We are using the below configuration setting in fluentd.conf to send parser apache access log to elasticsearch

<source>
  @type tail
  path /var/log/apache2/access.log 
  pos_file /var/log/td-agent/apache-access.log.pos
  tag apache.access
  format apache2
</source>

<filter apache.access>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

 <source>
   @type forward
   port 9200
 </source>

 <match *.*>
   @type elasticsearch
   logstash_format true
   host xx.xx.xx.xx 
   port 9200
   index_name fluentd 
   type_name fluentd
 </match>

but we the logs shown in kibana is not in parsed format. Please see the sample logs shown in kibana, accesslog_in_kiana_sample

We are expecting the logs in kibana similar to below,

_host: 127.0.0.1 - _user: - _time: [07/Sep/2017:04:52:19 +0000] _method: GET _path: /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" _code: 200 _size: 32775 _responsetime: 5113 _domain: "domain.test.com" _url: http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" _agent: "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"

We also tried by adding the below regex in fluentd.conf file in format,

format /^(?<host>[^ ]*(?:,\s+[^ ]+)*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) (?<responsetime>[^ ]*) "(?<domain>[^\"]*)" "(?<url>[^\"]*)" "(?<agent>[^\"]*)" "(?<jsession>[^\"]*)" "(?<skab>[^\"]*)" "(?<desktopview>[^\"]*)"/

I can't figure out what's wrong there above

0

There are 0 best solutions below