I have deployed Airbyte in EC2 using the instructions here My next goal is to send the logs to CloudWatch because I read somewhere that storing logs in EC2 has crashed the instance.
My EC2 details:
- In a custom VPC
- Private Subnet
- IAM Role attached to get full access to CloudWatch Logs
Please advise!
I have tried the following :
- IAM role created and attached it to my EC2 instance.
- CloudWatch Log Group
airbyte-ec2-DockerContainer
has been created - I have found out the command to configure the log group from here
- I did a
docker images -a
in my ec2, it returned all the images of airbyte services (airbyte/airbyte-api-server, airbyte/worker, airbyte/db etc) - Now, Airbyte comes with a startup script to up all the Airbyte services using
./run-ab-platform.sh -b
which I have been using to start Airbyte
How do I configure the logs to a CloudWatch Log Group in my case? It looks like the commands to do that are -
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/airbyte-api-server:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/worker:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/webapp:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/temporal:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/server:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/proxy:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/db:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/connector-builder-server:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/cron:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/init:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/bootloader:0.50.30
docker run -it --log-driver=awslogs --log-opt awslogs-region=us-east-1 --log-opt awslogs-group=airbyte-ec2-DockerContainer airbyte/source-mysql:3.0.8
But I haven't figured out where to run these commands since there already is a ./run-ab-platform.sh
script. How do I modify that script?