Can someone help me for Apache access log parsing on fluentd?
We are using Kibana to view the logs, but we can't able to split the message part using parser. below is our sample access log,
Sample Apache access log
127.0.0.1 - - [07/Sep/2017:04:52:19 +0000] "GET /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" 200 32775 5113 "domain.test.com" "http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"
We are using the below configuration setting in fluentd.conf to send parser apache access log to elasticsearch
<source>
@type tail
path /var/log/apache2/access.log
pos_file /var/log/td-agent/apache-access.log.pos
tag apache.access
format apache2
</source>
<filter apache.access>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
</record>
</filter>
<source>
@type forward
port 9200
</source>
<match *.*>
@type elasticsearch
logstash_format true
host xx.xx.xx.xx
port 9200
index_name fluentd
type_name fluentd
</match>
but we the logs shown in kibana is not in parsed format. Please see the sample logs shown in kibana, accesslog_in_kiana_sample
We are expecting the logs in kibana similar to below,
_host: 127.0.0.1 - _user: - _time: [07/Sep/2017:04:52:19 +0000] _method: GET _path: /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" _code: 200 _size: 32775 _responsetime: 5113 _domain: "domain.test.com" _url: http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" _agent: "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"
We also tried by adding the below regex in fluentd.conf file in format,
format /^(?<host>[^ ]*(?:,\s+[^ ]+)*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) (?<responsetime>[^ ]*) "(?<domain>[^\"]*)" "(?<url>[^\"]*)" "(?<agent>[^\"]*)" "(?<jsession>[^\"]*)" "(?<skab>[^\"]*)" "(?<desktopview>[^\"]*)"/
I can't figure out what's wrong there above