I have some 'generic' methods that extract data based on css selectors that usually are the same in many websites. However I have another method that accept as argument the css selector for a given website.
I need to call the get_title method if title_selector argument is nos passed. How can I do that?
Scrape that accept css selectors as arguments
def scrape(urls, item_selector, title_selector, price_selector, image_selector)
collection = []
urls.each do |url|
doc = Nokogiri::HTML(open(url).read) # Opens URL
@items = doc.css(item_selector)[0..1].map {|item| item['href']} # Sets items
@items.each do |item| # Donwload each link and parse
page = Nokogiri::HTML(open(item).read)
collection << {
:title => page.css(title_selector).text, # I guess I need conditional here
:price => page.css(price_selector).text
}
end
@collection = collection
end
end
Generic title extractor
def get_title(doc)
if doc.at_css("meta[property='og:title']")
title = doc.css("meta[property='og:title']")
else doc.css('title')
title = doc.at_css('title').text
end
end
Use an
or
operator inside yourpage.css
call. It will callget_title
iftitle_selector
is falsey (nil).I'm not sure what
doc
should actually be in this context, though.EDIT
Given your comment below, I think you can just refactor
get_title
to handle all of the logic. Allowget_title
to take an optionaltitle_selector
parameter and add this line to the top of your method:Then, my original line becomes: