Logging Databricks notebook to AppInsights in Python using Open Telemetry

Question

Our databricks notebook is trigerred via an adf pipeline. I would like to add logging in my python notebook and would like to connect that logging information to be viewed in appinsights. I would like to use the Open Telemetry API i.e. azure-monitor-opentelemetry. How do I achieve that in python? Any pointers code examples would help

Answer 1

To add logging to your Databricks notebook and connect it to AppInsights using OpenTelemetry, you can check Microsoft OpenTelemetry exporter for Azure Monitor

pip install azure-monitor-opentelemetry-exporter --pre

Below is example code snippet you can refer to configure your own python code logging.

import os
import logging

from opentelemetry.sdk._logs import (
    LoggerProvider,
    LoggingHandler,
    set_logger_provider,
)
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor

from azure.monitor.opentelemetry.exporter import AzureMonitorLogExporter

logger_provider = LoggerProvider()
set_logger_provider(logger_provider)

exporter = AzureMonitorLogExporter(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
handler = LoggingHandler()

# Attach LoggingHandler to root logger
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.NOTSET)

logger = logging.getLogger(__name__)
logger.warning("Hello World!")
logger_provider.force_flush()

For more details and other examples you can check this documentation.