Indexing 7TB of data with elasticsearch. FScrawler stops after sometime

Question

I am using fscrawler to create an index of data above 7TB. The indexing starts fine but then stops when the index size gets to 2.6gb. I believe this is a memory issue, how do I configure the memory?

My machine memory is 40GB and I have assigned 12GB to elasticsearch.

Answer 1

You might have to assign as well enough memory to FSCrawler using FS_JAVA_OPTS. Like:

FS_JAVA_OPTS="-Xmx4g -Xms4g" bin/fscrawler