Unable to connect elasticsearch from .NET Web API

Question

I'm trying to set up logging for my .NET Web API application using Serilog, and send logs to Elasticsearch and view them from Kibana. But I am receiving errors while trying to connect elasticsearch from API.

This is my Program.cs configuration to setup the logs

Log.Logger = new LoggerConfiguration()
    .ReadFrom.Configuration(builder.Configuration)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithEnvironmentName()
    .Enrich.WithExceptionDetails()
    .WriteTo.Console()
    .WriteTo.Elasticsearch(ConfigureElasticSink(builder.Configuration))                
    .CreateLogger();

builder.Host.UseSerilog();

private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration)
{
    var elasticUri = configuration.GetValue<string>("ElasticConfiguration:Uri");
    
    return new ElasticsearchSinkOptions(elasticUri != null ? new Uri(elasticUri) : null)
    {
        AutoRegisterTemplate = true,
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name?.ToLower().Replace(".","-")}-{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}-{DateTime.UtcNow:yyyy-MM--dd}",
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        NumberOfReplicas = 1,
        NumberOfShards= 2,
        OverwriteTemplate = true,
        TemplateName = "DevProject",
        TypeName = null,
        BatchAction = ElasticOpType.Create
    };            
}

I'm utilizing Serilog.Sinks.Elasticsearch package to post the logs. Elasticsearch and Kibana are both running in Docker.

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.1
    container_name: elasticsearch
    restart: always
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false 
    ports:
      - 9200:9200
    networks:
      - infrastructure
    volumes:
      - esdata:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.1
    container_name: kibana
    restart: always
    environment:
      - ELASTICSEARCH_URL=http://127.0.0.1:9200
    ports:
      - "5601:5601"
    networks:
      - infrastructure
    depends_on:
      - elasticsearch

volumes:
  esdata:

networks:
   infrastructure:
      name: infrastructure
      driver: bridge 

I've confirmed that Elasticsearch is healthy and operational.

{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 27,
  "active_shards" : 27,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

However, I'm encountering issues connecting to Elasticsearch from the .NET Web API. Below is a sample code snippet I'm using to test the Elasticsearch connection.

[HttpGet("TestConnection")]
public async Task<IActionResult> TestConnection()
{
    var connectionSettings = new ConnectionConfiguration(_elasticsearchUri);
    var lowLevelClient = new ElasticLowLevelClient(connectionSettings);

    try
    {
        var response = await lowLevelClient.PingAsync<StringResponse>();

        // Check if the ping was successful
        if (response.Success)
        {
            return Ok("Elasticsearch connection successful.");
        }
        else
        {
            return StatusCode(500, "Failed to connect to Elasticsearch.");
        }
    }
    catch (Exception ex)
    {
        return StatusCode(500, $"Error connecting to Elasticsearch: {ex.Message}");
    }
}

From this code I came across different errors by switching the nuget package version for Serilog.Sinks.ElasticSearch (errors encountered below) Error seen when using v10.0.0 (lastest stable version)

Unsuccessful () low level call on HEAD: /
# Audit trail of this API call:
 - [1] ProductCheckOnStartup: Took: 00:00:00.1923665
 - [2] ProductCheckFailure: Node: http://127.0.0.1:9200/ Took: 00:00:00.1923337
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. Call: Status code unknown from: GET /
 ---> Elasticsearch.Net.PipelineException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product.

Error seen when using v8.4.1

Unsuccessful () low level call on HEAD: /
# Audit trail of this API call:
 - [1] BadRequest: Node: http://127.0.0.1:9200/ Took: 00:00:00.0692046
# OriginalException: System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:9200)
 ---> System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---```

what am I missing?

Answer 1

Can you ping it with curl localhost:9200 in the terminal?

Try to use http://localhost:9200 instead as the URI, I think I've had this issue before.

This here is my setup https://www.zenitk.com/es-docker-compose-up