I managed to send my Apache access_log (httpd / amazon-linux-ami 2016.09 / php5.6) to elasticsearch via kinesis-agent and firehose.
Now I'm struggling to forward the error_log.
My agent.json
:
{
"cloudwatch.emitMetrics": true,
"kinesis.endpoint": "",
"firehose.endpoint": "firehose.eu-west-1.amazonaws.com",
"flows": [{
"filePattern": "/var/log/httpd/error_log*",
"deliveryStream": "xxxx",
"dataProcessingOptions": [{
"optionName": "LOGTOJSON",
"logFormat": "APACHEERRORLOG"
}]
}, {
"filePattern": "/var/log/httpd/access_log*",
"deliveryStream": "xxxx",
"dataProcessingOptions": [{
"optionName": "LOGTOJSON",
"logFormat": "COMMONAPACHELOG"
}]
}]
}
A sample line from /var/log/httpd/error_log
:
[Thu Feb 16 16:56:19.240650 2017] [:error] [pid 10719] [client 1.1.1.1:60890] PHP Notice: Undefined variable: env in /var/www/html/development/index.php on line 34
As I said, all access_logs are forwarded correctly. But no error logs are sent to destination. See below from /var/log/aws-kinesis-agent/aws-kinesis-agent.log
:
2017-02-17 06:32:05.486+0000 ip-172-31-7-92 (FileTailer[fh:xxx:/var/log/httpd/error_log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:xxx:/var/log/httpd/error_log*]: Tailer Progress: Tailer has parsed 414 records (116864 bytes), transformed 0 records, skipped 414 records, and has successfully sent 0 records to destination.
xxx
is my firehose destination (same for both).
To me it seems that my apache error logs (default format, I didn't change anything) are according to the format expected by APACHEERRORLOG.
What am I missing here? I would extremely appreciate some pointers, thanks a lot!