ELK Community Beats to gets AWS logs on ELK

Question

I have my ELK server setup on Ubuntu 16.xx (AWS EC2 instance). 1 - I need to monitor my AWS services (SNS, SQS, SES, Lambda, Ec2, S3 ELB CloudWatch .. etc.) 2 - how can I get the real-time logs from my aws service and post to my Kibana dashboard (is there any plugin/beat/integration api). are there community beats can help me to get the logs from AWS to ELK.

https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

Thanks

Answer 1

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-cloudwatch.html

You probably want to use the Cloudwatch Logstash plugin and config it to pull that data into your elasticsearch, I believe is that is the easiest way to do this.

Answer 2

A few more options to stream AWS logs to the ELK cluster are:

1) First push all your AWS logs to a S3 bucket and then use Logstash's "S3 input plugin" to stream logs from S3 to logstash.

2) Use a lambda function to stream all the cloudwatch logs directly to your ELK cluster. refer this: https://github.com/blueimp/aws-lambda/tree/master/cloudwatch-logs-to-elastic-cloud

Answer 3

You can use the below beats to retrieve cloudwatch logs and metrics.

cloudwatchmetricbeat - A beat for Amazon Web Services' CloudWatch Metrics.

cloudwatchlogsbeat - Reads log events from Amazon Web Services' CloudWatch Logs.

Answer 4

The Elastic Stack 6.5 has been released on Nov 14th and a new beat called Functionbeat allows you to stream your Cloudwatch logs easily to ES.