Hpricot - UTF-8 issues

Question

I get the following error when running the code below:

invalid byte sequence in UTF-8 (ArgumentError)

The code:

require 'hpricot'
require 'open-uri'

doc = open('http://www.amazon.co.jp/') {|f| Hpricot(f.read) }
puts doc.to_html

Hpricot cannot parse the Japanese content. Any suggestions on fixing this issue?

Answer 1

The site doesn't seem to be using UTF-8: <meta http-equiv="content-type" content="text/html; charset=Shift_JIS" />.

Try this instead:

open('http://www.amazon.co.jp/') {|f| Hpricot(f.read.encode("UTF-8")) }