Lucene search ton of names

Question

I was trying to Search ton of names(10000+) against lucene index, the names were loaded from a text file. This is snippet of my code:

Analyzer analyzer = new StandardAnalyzer();
MultiFieldQueryParser mParser = new MultiFieldQueryParser(arrSearchFields,
  analyzer);

Query keyWordsQuery = mParser.parse(names);

- First I get the error: too many boolean clauses at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:118)

as search on the internet, I can fix the by

BooleanQuery.setMaxClauseCount(Integer.MAX_VALUE);

But the search is slow and use a lot of memory.

Any suggestions for this case?

Appreciate it.

James

Answer 1

I know this is old but I hope someone finds this useful.

With Lucene 8.6.1, you can avoid "Too Many Clauses" exception by using the below query

(field1: term1 OR term2 OR .... OR term1024) OR (field1: term1025 OR trem1026 OR .... OR term2000)

instead of

field1: term1 OR term2 OR term3 OR .... OR term2000

Basically, just break up your query to consist no more than 1024 terms for each field at a time.

I'm not sure about memory usage or how fast this going to be but my intention was to get it working without throwing an error.

You may also want to look at this, this and this.