Authentication failed for user [admin] trying to access the Airflow APIs

Question

We have installed OpenMetadata instance shipped with Elastic and Airflow. We used official helm charts with their image "docker.getcollate.io/openmetadata/ingestion". I upgraded to newest helm chart version v1.1.4 and OpenMetadata version.

But we're facing a problem with connection from OpenMetadata to Airflow API. We set up a basic auth according docs (we have also tested a SSO KeyCloak which was working, but getting same issue ).

When I try add service - Pipeline or Database I got error (screenshot attached below): OpenMetadata error

OpenMetadata error

I read all available docs and update our configuration as official docs said:

values.yaml for dependencies chart:

airflow:
  enabled: {{ is_airflow_enabled }}
  airflow:
    image:
      repository: docker.getcollate.io/openmetadata/ingestion
      tag: {{ airflow_image_tag }}
      pullPolicy: "IfNotPresent"
    executor: "KubernetesExecutor"
    config:
      # This is required for OpenMetadata UI to fetch status of DAGs
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth"
      # OpenMetadata Airflow Apis Plugin DAGs Configuration
      AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/opt/airflow/dags"
      # OpenMetadata Airflow Secrets Manager Configuration
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_REGION: ""
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY_ID: ""
      AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AWS_ACCESS_KEY: ""
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'true'

    users:
    - username: admin
      password: admin
      role: Admin
      email: [email protected]
      firstName: Peter
      lastName: Parker

values.yaml for OpenMetadata chart:

    pipelineServiceClientConfig:
      enabled: true
      className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
      apiEndpoint: http://openmetadata-dependencies-web.{{ openmetadata_namespace }}.svc.cluster.local:8080
      metadataApiEndpoint: http://openmetadata.{{ openmetadata_namespace }}.svc.cluster.local:8585/api
      verifySsl: "no-ssl"
      hostIp: ""
      ingestionIpInfoEnabled: false
      # healthCheckInterval in seconds
      healthCheckInterval: 300
      # local path in Airflow Pod
      sslCertificatePath: "/no/path"
      auth:
        username: admin
        password:
          secretRef: airflow-secrets
          secretKey: openmetadata-airflow-password

I tried create different user but same error (created new secret, upgraded airlflow values.yaml file for new user and gave Admin role).I don't understand why I'm getting that error in browser, because when I jump to OpenMetadata pod and test connection to Airflow API all seems fine:

bash-5.1# curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health                                                                                                     
{"status": "healthy", "version": "1.1.0.3"}

Also when I tried call Airflow API with user, I didn't get error due to 401 Unauthorized:

bash-5.1# curl -XPOST ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/enable --data-raw '{"dag_id": "123"}' -u "admin:admin" --header 'Content-Type: application/json'
{"error": "DAG 123 not found."}

I use default user admin/admin and I am able to login to Airflow UI.

when I call env variables from OpenMetadata container, it seems fine: OM env varaibles for Airflow

From Openmetadata pod, I can call Airflow API with credentials:

bash-5.1# curl -XGET ${PIPELINE_SERVICE_CLIENT_ENDPOINT}/api/v1/openmetadata/health-auth -u "admin:admin"
{"status": "healthy", "version": "1.1.0.3"}