write log to Application insights from local service fabric

Question

I am trying to integrate Azure App insights service into the service fabric app for logging and instrumentation. I am running fabric code on my local VM. I exactly followed the document here [scenario 2]. Other resources on learn.microsoft.com also seem to indicate the same steps. [ex: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-aggregation-eventflow For some reason, I don’t see any event entries in App insights. No errors in code when I do this:

ServiceEventSource.Current.ProcessedCountMetric("synced",sw.ElapsedMilliseconds, crc.DataTable.Rows.Count);

eventflowconfig.json contents

    {
      "inputs": [
        {
          "type": "EventSource",
          "sources": [
            { "providerName": "Microsoft-ServiceFabric-Services" },
            { "providerName": "Microsoft-ServiceFabric-Actors" },        
            { "providerName": "mystatefulservice" }
          ]
        }
      ],
      "filters": [
        {
          "type": "drop",
          "include": "Level == Verbose"
        }
      ],
      "outputs": [
        {
          "type": "ApplicationInsights",
          // (replace the following value with your AI resource's instrumentation key)
          "instrumentationKey": "XXXXXXXXXXXXXXXXXXXXXX",
          "filters": [
            {
              "type": "metadata",
              "metadata": "metric",
              "include": "ProviderName == mystatefulservice && EventName == ProcessedCountMetric",
              "operationProperty": "operation",
              "elapsedMilliSecondsProperty": "elapsedMilliSeconds",
              "recordCountProperty": "recordCount"
            }
          ]
        }
      ],
      "schemaVersion": "2016-08-11"
    }

In ServiceEventSource.cs

[Event(ProcessedCountMetricEventId, Level = EventLevel.Informational)]
    public void ProcessedCountMetric(string operation, long     elapsedMilliSeconds, int recordCount)
    {
        if (IsEnabled())
            WriteEvent(ProcessedCountMetricEventId, operation, elapsedMilliSeconds, recordCount);
    }

EDIT Adding diagnostics pipeline code from "Program.cs in fabric stateful service

using (var diagnosticsPipeline =
                ServiceFabricDiagnosticPipelineFactory.CreatePipeline($"{ServiceFabricGlobalConstants.AppName}-mystatefulservice-DiagnosticsPipeline")
            )
            {
                ServiceRuntime.RegisterServiceAsync("mystatefulserviceType",
                    context => new mystatefulservice(context)).GetAwaiter().GetResult();

                ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id,
                    typeof(mystatefulservice).Name);

                // Prevents this host process from terminating so services keep running.
                Thread.Sleep(Timeout.Infinite);
            }

Answer 1

Event Source is a tricky technology, I have been working with it for a while and always have problems. The configuration looks good, It is very hard to investigate without access to the environments, so I will make my suggestions.

There are a few catches you must be aware of:

• If you are listening etw events from a different process, your process must be running with a user with permissions on 'Performance Log Users. Check which identity your service is running on and if it is part of performance log users, who has permissions to create event sessions to listen for these events.

• Ensure the events are being emitted correctly and you can listen to them from Diagnostics Events Window, if it is not showing there, there is a problem in the provider.

• For testing purposes, comment out the line if (IsEnabled()). it is an internal check to validate if your events should be emitted. I had situations where it is always false and skip the emit of events, probably it cache the results for a while, the docs are not clear how it should work.

• Whenever possible, use the EventSource from the nuget package instead of the framework one, the framework version is full of bugs and lack fixes found in the nuget version.

• Application Insights are not RealTime, sometimes it might take a few minutes to process your events, I would recommend to output the events to a console or file and check if it is listening correctly, afterwards, enable the AppInsights.

Answer 2

The link you provide is quite outdated and there's actually a much better way to log application error and exception info to Application insights. For example, the above won't help with tracking the call hierarchy of an incoming request between multiple services.

Have a look at the Microsoft App Insights Service Fabric nuget packages. It works great for:

• Sending error and exception info
• Populating the application map with all your services and their dependencies (including database)
• Reporting on app performance metrics,
• Tracing service call dependencies end-to-end,
• Integrating with native as well as non-native SF applications