Storing percolator in separate index when only one mapping is possible per index?

Question

I have a SearchAgent document in an index called searchagent which looks like this:

[ElasticsearchType(IdProperty = "Id")]
public class SearchAgent
{
    public string Id { get; set; }

    [Keyword]
    public string UserId { get; set; }

    public QueryContainer Query { get; set; }
}

This is because I want my users to create "search agents" that will notify the user when a new document for a specific search has been inserted.

Now, the document that I want to find the relevant search agent for, is in the items index and is an Item. It looks like the following:

[ElasticsearchType(IdProperty = "Id")]
public class Item
{
    public string Id { get; set; }
    public string Title { get; set; }
}

This also seems to be what the documentation recommends:

Given the design of percolation, it often makes sense to use separate indices for the percolate queries and documents being percolated, as opposed to a single index ...

However, I can't index my search agent documents now, since their Query refers to a property on the Item document. And this leads to the following error:

No field mapping can be found for the field with name [title]

I guess this means that I both have to describe the Item and SearchAgent mapping in the searchagent index.

But in Elasticsearch 6, they removed support for multiple mappings per index, so this is not possible.

How can I get around this problem?

Answer 1

I guess this means that I both have to describe the Item and SearchAgent mapping in the searchagent index.

This is correct for 6.x. Essentially, percolation needs to know about the mapping of the documents that will be percolated, so the index that contains the queries also needs to have the fields for the documents that will be percolated.

With NEST 6.x, it's possible to do this with

var client = new ElasticClient();

var createIndexResponse = client.CreateIndex("percolation", c => c
    .Mappings(m => m
        .Map<SearchAgent>(mm => mm
            .AutoMap<SearchAgent>()
            .AutoMap<Item>()
        )
    )
);

This will automap the properties of both SearchAgent and Item under the mapping for SearchAgent and will result in the following request

PUT http://localhost:9200/percolation?pretty=true 
{
  "mappings": {
    "searchagent": {
      "properties": {
        "id": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "title": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "userId": {
          "type": "keyword"
        },
        "query": {
          "type": "percolator"
        }
      }
    }
  }
}

NOTE that a property with the same name on both POCOs will take the mapping of the last property of that name to be mapped, so it's recommended that the properties have the same mapping or better yet, that the query documents contain differently named properties (Id is fine if they're both mapped the same), to avoid potential confusion down the track.

With the percolation index set up, targeting documents in another index would now be achieved with

var searchResponse = client.Search<SearchAgent>(s => s
    .Index("percolation") // index containing queries
    .Query(q => q
        .Percolate(p => p
            .Type<Item>() 
            .Index("items") // index containing documents
            .Id("item-id") // document id
            .Field(f => f.Query) // field on SearchAgent containing query
        )        
    )
);

which executes the following query

POST http://localhost:9200/percolation/searchagent/_search 
{
  "query": {
    "percolate": {
      "field": "query",
      "id": "item-id",
      "index": "items",
      "type": "item"
    }
  }
}

You may want to also set up conventions for the POCOs on ConnectionSettings

var defaultIndex = "default-index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));

var settings = new ConnectionSettings(pool)
    .DefaultIndex(defaultIndex)
    .DefaultMappingFor<SearchAgent>(d => d
        .IndexName("percolation")
    )
    .DefaultMappingFor<Item>(d => d
        .IndexName("items")
    );

var client = new ElasticClient(settings);

Then the search request can use

var searchResponse = client.Search<SearchAgent>(s => s
    .Query(q => q
        .Percolate(p => p
            .Type<Item>()
            .Index<Item>()
            .Id("item-id")
            .Field(f => f.Query)
        )        
    )
);

Finally, take a look at the NEST Percolation Query DSL documentation; it could definitely do with improvement as it omits some information that is not included when it is autogenerated from the test suite, but hopefully it gives you an idea. With any NEST documentation, you can always click the edit button on the docs page to get a link to the original source from which it is generated:

and

which leads to https://github.com/elastic/elasticsearch-net/blob/6.x/src/Tests/Tests/QueryDsl/Specialized/Percolate/PercolateQueryUsageTests.cs in the case of the Percolate 6.x documentation.