Sent APACHEERRORLOG to Elasticsearch via amazon-kinesis-agent

193 Views Asked by At

I managed to send my Apache access_log (httpd / amazon-linux-ami 2016.09 / php5.6) to elasticsearch via kinesis-agent and firehose.

Now I'm struggling to forward the error_log.

My agent.json:

{
        "cloudwatch.emitMetrics": true,
        "kinesis.endpoint": "",
        "firehose.endpoint": "firehose.eu-west-1.amazonaws.com",
        "flows": [{
                "filePattern": "/var/log/httpd/error_log*",
                "deliveryStream": "xxxx",
                "dataProcessingOptions": [{
                        "optionName": "LOGTOJSON",
                        "logFormat": "APACHEERRORLOG"
                }]
        }, {
                "filePattern": "/var/log/httpd/access_log*",
                "deliveryStream": "xxxx",
                "dataProcessingOptions": [{
                        "optionName": "LOGTOJSON",
                        "logFormat": "COMMONAPACHELOG"
                }]
        }]
}

A sample line from /var/log/httpd/error_log:

[Thu Feb 16 16:56:19.240650 2017] [:error] [pid 10719] [client 1.1.1.1:60890] PHP Notice:  Undefined variable: env in /var/www/html/development/index.php on line 34

As I said, all access_logs are forwarded correctly. But no error logs are sent to destination. See below from /var/log/aws-kinesis-agent/aws-kinesis-agent.log:

2017-02-17 06:32:05.486+0000 ip-172-31-7-92 (FileTailer[fh:xxx:/var/log/httpd/error_log*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:xxx:/var/log/httpd/error_log*]: Tailer Progress: Tailer has parsed 414 records (116864 bytes), transformed 0 records, skipped 414 records, and has successfully sent 0 records to destination.

xxx is my firehose destination (same for both).

To me it seems that my apache error logs (default format, I didn't change anything) are according to the format expected by APACHEERRORLOG.

What am I missing here? I would extremely appreciate some pointers, thanks a lot!

0

There are 0 best solutions below