in Fscrawler is it possible to derive index name from subfolder name?

Question

I have a folder with several hundred sub folders and would like to index by these sub folder names...That is after running fscrawler I would like to have several hundred different indices...The default is one index (e.g."job_name") for all the sub directories...Is it possible to create indexes for each individual sub folder?

Answer 1

Two options I can think of:

• One job per folder
• Using an ingest pipeline where you can overwrite the index name with a set processor.

Answer 2

Instead of using the set processor I did it using aliases in python where output is a list of the folder names I derived from _source.path.virtual:

for x in output:
  es.indices.update_aliases({
  actions": [
  { "add":    { "index": 'job_old', "alias": x,"filter":{"regexp":{"path.virtual.keyword":"/"+x+".*"}}}}
    
      ]
    })

The convert each alias to custom index.....