3) do |anemone| anemone.on_every_page do |" /> 3) do |anemone| anemone.on_every_page do |" /> 3) do |anemone| anemone.on_every_page do |"/>

Matching URL structures with Anemone

117 Views Asked by At

Right now, I'm doing the following with Anemone:

Anemone.crawl("http://www.findbrowsenodes.com/", :delay => 3) do |anemone|
  anemone.on_every_page do | page |

But I would like to do    

Anemone.crawl("http://www.findbrowsenodes.com/", :delay => 3) do |anemone|
   anemone.on_pages_like() do | page |

instead to only crawl from URLs like this:

http://www.findbrowsenodes.com/us/Apparel/1036682 http://www.findbrowsenodes.com/us/Apparel/1040664 http://www.findbrowsenodes.com/us/ArtsAndCrafts/2617942011 http://www.findbrowsenodes.com/us/Baby/165797011

Any ideas how?

1

There are 1 best solutions below

1
rhernando On

you can use a regular expression like this

/http:\/\/www.findbrowsenodes.com\/us\/.+\/[\d]*/