Emit a custom metadata from seed URLs through all child discovered URLs for all depth

80 Views Asked by At

I have a Storm Crawler based project which index all contents and status in Solr collections. For each seedUrl, I have some metadata which needs to be emitted through all child of each seed URLs. For example, I have a data structure similar to this:

<crawlId, seedUrl, myMetadata>

How can I emit the crawlId and corresponding metaData to all discovered children for each seedUrl? Is there available any built-in capability that we can use from that or not?

1

There are 1 best solutions below

0
On BEST ANSWER

metadata.transfer is what you need, see conf from the archetype