Can't an index with stopFilter and synonymFilter be created in ES 6.8?

99 Views Asked by At

I have an index will following settings in ES 2.4.

{"settings":{"index":{"analysis":{"filter":{"stopFilter":{"type":"stop","remove_trailing":"false","stopwords":"_english_"},"stemmerFilter":{"type":"stemmer","language":"light_english"},"synonymFilter":{"type":"synonym","synonyms":["3pip, third party checkout with immediate => 3PIP","b to b, business to business, btob, b2b, business 2 business, b 2 b => B2B","b 2 e, business to employee, biz to employee, btoe, b to e, b2e => B2E","b 2 g, b to g, btog, b2g => B2G"]},"possessiveFilter":{"type":"stemmer","language":"possessive_english"}},"char_filter":{"quotes":{"type":"mapping","mappings":["\\u0091=>\\u0027","\\u0092=>\\u0027","\\u2018=>\\u0027","\\u2019=>\\u0027","\\u201B=>\\u0027"]}},"analyzer":{"default":{"filter":["possessiveFilter","lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"char_filter":["html_strip","quotes"],"type":"custom","tokenizer":"standard"},"lowercase":{"filter":["lowercase"],"type":"custom","tokenizer":"keyword"},"autocomplete":{"filter":["possessiveFilter","lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"char_filter":["html_strip","quotes"],"type":"custom","tokenizer":"autocompleteTokenizer"},"autocomplete_search":{"filter":["lowercase","stopFilter","stemmerFilter","synonymFilter","asciifolding"],"tokenizer":"standard"}},"tokenizer":{"autocompleteTokenizer":{"token_chars":["letter","digit"],"min_gram":"2","type":"edgeNGram","max_gram":"20"}}}}}}

But index creation with same settings fail in ES 6.8 with errors like **"type":"illegal_argument_exception","reason":"failed to build synonyms","caused_by":{"type":"parse_exception","reason":"parse_exception: Invalid synonym rule at line 5","caused_by":{"type":"illegal_argument_exception","reason":"term: a b a analyzed to a token (b) with position increment != 1 (got: 2)"}}},"status":400}

and

,"reason":"term: third party checkout with immediate analyzed to a token (immediate) with position increment != 1 (got: 2)"**

Index gets created after removing these words but that's not an option.

Is there a way to create the index with the same settings (stopFilter and synonymFilter together) as in ES 2.4?

https://github.com/elastic/elasticsearch/issues/28838#issuecomment-629115702 suggests that this is a bug and stopFilter and synonymFilter cannot be used together.

1

There are 1 best solutions below

1
rabbitbr On

Reading about this problem I found this post, in some cases using lenient:true solved the problem.

"synonymFilter": {
            "type": "synonym",
            "lenient":true,
            "synonyms": [
              "3pip, third party checkout with immediate => 3PIP",
              "b to b, business to business, btob, b2b, business 2 business, b 2 b => B2B",
              "b 2 e, business to employee, biz to employee, btoe, b to e, b2e => B2E",
              "b 2 g, b to g, btog, b2g => B2G"
            ]
          }