I have some stderr files(logs) stored in an azure blob that are taken from databricks dbfs, how can I send these log messages into azure monitor /log analytics workspace in same subscription.
I have tried following microsoft documentations, but couldnt find one that could actually help me solve this issue.
You can send the logs that come from Azure Databricks and are saved in Azure Data Lake Storage to Log Analytics workspace. One approach to accomplish this is by using the Azure Log Analytics HTTP Data Collector API.
Sencond approach is using the Logs Ingestion API in Azure Monitor
The below are the Configuration Steps:
Step 1: Create a Microsoft Entra application and service principal that can access resources and also Create a new application secret.
In the Access control (IAM) Add Contributor role. for your SPN.
Step 2: Create a data collection endpoint.
Here see an example to how send data to Azure Monitor using Logs ingestion API.
Step 3: Create a custom Table in Log Analytics workspace
Step 4: create DCR Data collection rule (DCR)
You can also send Azure Databricks application logs to Azure Monitor
References: Create a Microsoft Entra application and service principal that can access resources