Bulk insert into elasticsearch using Elasticsearch.net or PlainElastic.net

Question

I've being messing about with both Elasticsearch.net (http://nest.azurewebsites.net/) and PlainElastic.Net(https://github.com/Yegoroff/PlainElastic.Net) and have been able to insert single documents into elasticsearch. I am now trying to figure out how to perform a bulk insert. I know each of these two .net libraries have documentation around this, but the data I wish to insert is stored in a dictionary where the key is the ID of the document and value is the document and I cannot figure out how to do it.

Here is some code I have (using Elasticsearch.net) :

var conn = new Uri("http://localhost:9200");
var config = new ConnectionConfiguration(conn);
var client = new ElasticsearchClient(config);

var myJson = @"{""Col1"" : ""Hello World"", ""col2"" : ""asdfasdf"" }";
var myjson2 = @"{""Col2"" : ""Hello World Again"", ""col2"" : ""zxcvzxcv"" }";

Dictionary<string, string> jsonCollection = new Dictionary<string, string>();

jsonCollection.Add("1", myJson);
jsonCollection.Add("2", myjson2);

Answer 1

I use PlainElastic.Net and this is how RAW data should look like

POST /_bulk
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 1  }}
{ "id": 1, "name": "My category \"ONE\" "}
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 2  }}
{ "id": 2, "name": "My second category \t "}
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 3 }}
{ "id": 3, "name": "My third category \r\n "}

keep in mind that new line is at the end of every row (yes even after last line)

so vb.net should look like this:

Dim bulkData As String = "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 1  }}" & vbNewLine & _
                         "{ ""id"": 1, ""name"": ""My category""}" & vbNewLine & _
                         "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 2  }}" & vbNewLine & _
                         "{ ""id"": 2, ""name"": ""My second category""} " & vbNewLine & _
                         "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 3  }}" & vbNewLine & _
                         "{ ""id"": 3, ""name"": ""My third category""} " & vbNewLine

Dim ESConn as New ElasticConnection("localhost", 9200)
Dim response As String = ESConn.Post("/_bulk", bulkData)

c# version I didn't test but you'll get the idea

string bulkData = @"{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 1}}
{ ""id"": 1, ""name"": ""My category""}
{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 2}}
{ ""id"": 2, ""name"": ""My second category""}
{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 3}}
{ ""id"": 3, ""name"": ""My third category""} \n";

ElasticConnection ESConn = New ElasticConnection("localhost", 9200);
string response = ESConn.Post("/_bulk", bulkData);

You can create JSON manually or with Newtonsoft.Json

Answer 2

I am using below class for loading bulk indexes. this example don't use Json document.

public class esclient
{
    const string uri = "http://localhost:9200";
    const string index = "INDEX_NAME";
    static ElasticClient _current;
    public esclient()
    {
        if (_current == null)
        {
            var node = new Uri(uri);

            var _settings = new ConnectionSettings(node)
                            .DefaultIndex(index)
                            .MaximumRetries(2)
                            .MaxRetryTimeout(TimeSpan.FromSeconds(150));

            _current = new ElasticClient(_settings);
        }
    }

    public void bulkIndexCreate(IEnumerable<esentity> items)
    {
        var descriptor = new BulkDescriptor();
        foreach (var item in items)
        {
            descriptor.Index<esentity>(op => op.Document(item));
        }

        var result = _current.Bulk(descriptor);
    }
}