steeltoe out of memory

Question

There was a problem with the operation of the steeltoe. Previously, we did not have a load and this problem did not arise. At the moment, the service is under load of 270-300 requests per second. And within 3 hours memory is clogged. We use 3 replicas and each with 4 GB of RAM. After removing the memory dump, it turned out that everything was clogged with strings. With a more detailed breakdown, it turned out that metrics were being collected. We have connected 2 Info and health actuators. But at the same time, all other enpoints are available by default. In configuration

Configuration

eureka:
  instance:
    StatusPageUrlPath: "/actuator/info"
    HealthCheckUrlPath: "/actuator/health"

endpoints:
    actuator:
      exposure:
        include: [info, health]
        exclude: [cloudfoundry, dbmigrations, env, heapdump, httptrace, hypermedia, loggers, mappings, prometheus, refresh, threaddump, metrics]
    cloudfoundry:
      enabled: false
    dbmigrations:
      enabled: false
    env:
      enabled: false
    heapdump:
      enabled: false
    httptrace:
      enabled: false
    hypermedia:
      enabled: false
    loggers:
      enabled: false
    mappings:
      enabled: false
    metrics:
      enabled: false
    prometheus:
      enabled: false
    refresh:
      enabled: false
    threaddump:
      enabled: false

I’ll make a reservation right away that everything that is indicated in the config at the moment is already experiments. With these settings, enpoints such as "httptrace" are not available, but before the exception they were available and it was possible to directly see the data in the browser. It didn't solve the memory problem. When studying steeltoe sources, we found out that metrics are being cleared, but for this you need to connect "metrics" or "prometheus".

I would like to know why the metrics are saved if I specify specific 2 actuators? If I don't need this data, how can I customize the configuration without saving it?

Answer 1

The .AddSteeltoe method was created to simplify the Steeltoe configuration experience. It uses other pre-existing methods for wiring up Steeltoe components, but is only able to configure those which have been added via NuGet references. Exclusions are supported, because it isn't feasible to support passing parameters. Standard ways of configuring components via app configuration should not be affected by your choice of using .AddSteeltoe or the other methods.

It is true that there are some observability pieces (eg: HttpTraceDiagnosticObserver and AspNetCoreHostingObserver) that automatically start up even when related actuators are not enabled, but there are safeguards to avoid endless memory usage. I've just tested a sample app (with 50+ threads making continuous requests) like you'll see relevant bits of below, and the memory climbs until the buffers fill (like 85mb to 135mb) but after it stabilizes there's no further memory growth.

It is possible to enable/disable specific metrics observers if you are still concerned about their memory usage. I don't think there's currently a way to disable the http trace observer, so if you'd like to see that it would be a good feature request.

If this doesn't answer your question, I'd recommend opening an issue in the Steeltoe repo on github and including a sample that reproduces the issue.

Sample project file:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Steeltoe.Bootstrap.Autoconfig" Version="3.1.1" />
    <PackageReference Include="Steeltoe.Management.EndpointCore" Version="3.1.1" />
  </ItemGroup>
</Project>

Sample Program.cs

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;
using Steeltoe.Bootstrap.Autoconfig;

namespace WebApplication6
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                })
                .AddSteeltoe();
    }
}