Send docker container logs on EC2 to CloudWatch

113 Views Asked by At

I have deployed Airbyte in EC2 using the instructions here My next goal is to send the logs to CloudWatch because I read somewhere that storing logs in EC2 has crashed the instance.

My EC2 details:

  1. In a custom VPC
  2. Private Subnet
  3. IAM Role attached to get full access to CloudWatch Logs

Please advise!

I have tried the following :

  • IAM role created and attached it to my EC2 instance.
  • CloudWatch Log Group airbyte-ec2-DockerContainer has been created
  • I have found out the command to configure the log group from here
  • I did a docker images -a in my ec2, it returned all the images of airbyte services (airbyte/airbyte-api-server, airbyte/worker, airbyte/db etc)
  • Now, Airbyte comes with a startup script to up all the Airbyte services using ./run-ab-platform.sh -b which I have been using to start Airbyte

How do I configure the logs to a CloudWatch Log Group in my case? It looks like the commands to do that are -

docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/airbyte-api-server:0.50.30        
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/worker:0.50.30                    
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/webapp:0.50.30                   
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/temporal:0.50.30                 
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/server:0.50.30                    
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/proxy:0.50.30                    
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/db:0.50.30                       
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/connector-builder-server:0.50.30  
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/cron:0.50.30                      
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/init:0.50.30                      
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/bootloader:0.50.30                
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/source-mysql:3.0.8

But I haven't figured out where to run these commands since there already is a ./run-ab-platform.sh script. How do I modify that script?

0

There are 0 best solutions below