How to search AWS OpenSearch instance from .Net Core application C#

346 Views Asked by At

I have created an AWS Kinesis Data Firehose delivery stream, its destination is an AWS OpenSearch Domain. These are the config settings for both of them:

Open Search enter image description here

enter image description here

Data Firehose

enter image description here

On the .Net site, I created a dummy app, which uses Audit.Net and on the EmployeeContext which inherits from AuditDbContext I have this

 public override void OnScopeSaving(IAuditScope auditScope)
    {
        try
        {
            PutRecordRequest putRecordRequest = new PutRecordRequest
            {
                DeliveryStreamName = "PUT-OPS-Cuc7o"
            };

            string data = "test";
            MemoryStream stream = new MemoryStream();
            using (BinaryWriter writer = new BinaryWriter(stream))
            {
                writer.Write(data);
            }

            Record record = new Record
            {
                Data = stream
            };
            putRecordRequest.Record = record;
            _kinesisFirehose.PutRecordAsync(putRecordRequest).Wait();

        }
        catch (Exception ex)
        {
            throw;
        }
    }

As you can see right now I'm just sending "test" to the data stream and to the Open Search service, but the idea is to send the json with the entity changes and use AWS OpenSearch to store the audit logs and then be able to query that same service to search the logs.

Now, this is how I'm registering the AWS service for Firehose enter image description here

Please keep in mind this is just a spike I'm doing to see it it is possible, so is not production code, and also this is the first time I'm using AWS at all.

Now I've been searching the AWS documentation, and in google, but can't find an example on how I can connect to AWS Open search from my c# application in order to query the service.

I saw this How to do AWS signature authorization and invoke elastic search using .NET Core client, but I'm not sure what is the url that I should be using, I tried

var url = "https://vpc-audit-log-test-edbgtml6pboqr6a5njyfx4dh5a.us-east-2.es.amazonaws.com";
var defaultIndex = "logs-index";
var httpConnection = new AwsHttpConnection("us-east-2");

var pool = new SingleNodeConnectionPool(new Uri(url));
var config = new ConnectionSettings(pool, httpConnection);

config.DefaultIndex(defaultIndex);
config.DefaultFieldNameInferrer(p => p.Normalize());
config.DisableDirectStreaming();

var client = new ElasticClient(config);

builder.Services.AddSingleton(client);

But it fails, I have reviewed https://awswith.net/2021/10/01/getting-started-with-elasticsearch-amazon-opensearch-nest-and-net/ but that didn't worked for me either, the AWS SDK documentation doesn't have a .Net example, only mentions Java, and I'm not even sure there is an AWS SDK package for searching on OpenSearch, I'm currently using the ones mentioned on that other Stack Overflow question mentioned above, which are

<PackageReference Include="NEST" Version="7.12.1" />
<PackageReference Include="Elasticsearch.Net.Aws" Version="7.1.0" />

And I also enabled the backwards compatibility on the open search service in AWS.

So my question is if anyone knows or can point me to an example or tutorial on how can I connect to the AWS OpenSearch service to query and get data back from my c# application.

Also, something I should mention is, if I try to use the OpenSearch Dashboards URL that is shown on the AWS console enter image description here I get an ERR_CONNECTION_TIMED_OUT.

Thanks for the help,

0

There are 0 best solutions below