How to auto-index data using solr and nutch?

Question

i want to automatically index a document or a website when it is fed to apache solr . How we can achieve this ? I have seen examples of using a CRON job that need to be called via a php script , but they are not quite clear in explaination. Using java api SolrJ , is there any way that we can index data automatically , without having the need to manually do it ??

Answer 1

You can write a scheduler and call the solrJ code which is doing indexing/reindexing.

For writing the scheduler please refer below links

http://www.mkyong.com/java/how-to-run-a-task-periodically-in-java/

http://archive.oreilly.com/pub/a/java/archive/quartz.html

Answer 2

If you are using Apache Nutch, you have to use Nutch solr-index plugin. With using this plugin you can index web documents as soon as they be crawled by Nutch. But the main question would be how can you schedule Nutch to start periodically.

As far as I know you have to use a scheduler for this purpose. I did know an old Nutch project called Nutch-base which uses Apache Quartz for the purpose of scheduling Nutch jobs. You can find the source code of Nutch-base from the following link:

https://github.com/mathieuravaux/nutchbase

If you consider this project there is a plugin called admin-scheduling. Although it is implemented for and old version of Nutch but it could be a nice start point for developing scheduler plugin for Nutch.

It is worth to say that if you are going to crawl website periodically and fetch the new arrival links you can use this tutorial.