How to pull private docker image from GitLab container registry using DockerOperator in Airflow 2.0?

Question

I have installed Airflow 2.0 using Docker on AWS and trying to pull my own private docker image GitLab container registry using DockerOperator but getting permission denied.

How I can pull my docker image from GitLab and How I can create a connection for the Docker image GitLab registry?

Dag file link Docker yml file link

My dag Code:

java_test_scheduler = DockerOperator(
    task_id='Java-hello-world-test-scheduler',
    image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest',
    auto_remove=True,
    force_pull=True,
    dag=dag
)

I also tried by adding docker volums:

- /var/run/docker.sock:/var/run/docker.sock
- $HOME/.ssh:/root/.ssh:ro
- $HOME/.docker:/root/.docker

Error:

sock.connect(self.unix_socket) PermissionError: [Errno 13] Permission denied

Answer 1

Your dag is missing dag_conn_id definition when pulling from private docker registry:

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html

Answer 2

1. I have tried without docker you can check this answer for details on that
2. I wanted to keep least dependency on airflow and hence didn't create the connection from airflow whose details are mentioned in the above referenced answer link
3. As far as your dag link is concerned, I had tried similar approach but could not make it work with image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest', however it did work out when I removed the :latest and use it like image='registry.gitlab.com/mobinalhassan/jamay_aeronova',, try this out and let me know if it works
4. For full details on pulling private docker image, I have jotted down a step by step tutorial which you can refer at - this link