Issue with Airflow remote logs on Google Stackdriver

118 Views Asked by At

I have Airflow 2.5.1 deployed on a Kubernetes Cluster with the official helm chart. The executor is KubernetesExecutor.
I want to write and read the airflow logs remotely on Stackdriver.
In my code I followed the steps described in the official airflow documentation so I modified the airflow helm chart adding the following env variables:

# Environment variables for all airflow containers
env:
- name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
  value: "True"
- name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
  value: "stackdriver://airflow.task"
- name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
  value: "google_cloud_default" 

In Stackdriver I created the bucket "airflow.task".

In airflow, I have a dummy task like this:

from airflow.models import Variable
# import the logging module
import logging

# get the airflow.task logger
task_logger = logging.getLogger("airflow.task")

def print_hello_world(ds):
    # with default airflow logging settings, DEBUG logs are ignored
    task_logger.debug("This log is at the level of DEBUG")

    # each of these lines produces a log statement
    print("This log is created with a print statement")
    task_logger.info("This log is informational")
    task_logger.warning("This log is a warning")
    task_logger.error("This log shows an error!")
    task_logger.critical("This log shows a critical error!")

When I run the task I have two problems:

  1. Airflow's logs are written in the Stackdriver bucket "_Default" instead of "airflow.task"
  2. In the airflow UI and in Stackdriver I can see only build-in task logs, but not my custom logs (see example below).
[2023-04-18, 10:41:38 UTC] {standard_task_runner.py:55} INFO - Started process 36 to run task
[2023-04-18, 10:42:49 UTC] {local_task_job.py:276} WARNING - State of this instance has been externally set to success. Terminating instance.
[2023-04-18, 10:42:49 UTC] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 36. PIDs of all processes in the group: [36]
[2023-04-18, 10:42:49 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 36

Someone have any suggestion? I have check online for this problem, but I did not find much information.

0

There are 0 best solutions below