tackle different types of utf hyphens in ruby 1.8.7

Question

We have different types of hyphens/dashes (in some text) populated in db. Before comparing them with some user input text, i have to normalize any type of dashes/hyphens to simple hyphen/minus (ascii 45).

The possible dashes we have to convert are:

Minus(−) U+2212 − or − or −
Hyphen-minus(-) U+002D -
Hyphen(-) U+2010
Soft Hyphen   U+00AD  ­
Non-breaking hyphen  U+2011  &#8209
Figure dash(‒)  U+2012 (8210) ‒ or ‒
En dash(–) U+2013 (8211) –, – or –
Em dash(—) U+2014 (8212) —, — or —
Horizontal bar(―) U+2015 (8213) ― or ―

These all have to be converted to Hyphen-minus(-) using gsub. I've used CharDet gem to detect the character encoding type of the fetched string. It's showing windows-1252. I've tried Iconv to convert the encoding to ascii. But it's throwing an exception Iconv::IllegalSequence.

ruby -v => ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin9.8.0]
rails -v => Rails 2.3.5
mysql encoding => 'latin1'

Any idea how to accomplish this?

Answer 1

Caveat: I know nothing about Ruby, but you have problems that are nothing to do with the programming language that you are using.

You don't need to convert Hyphen-minus(-) U+002D - to simple hyphen/minus (ascii 45); they're the same thing.

You believe that the database encoding is latin1. The statement "My data is encoded in ISO-8859-1 aka latin1" is up there with "The check is in the mail" and "Of course I'll still love you in the morning". All it tells you is that it is a single-byte-per-character encoding.

Presuming that "fetched string" means "byte string extracted from the database", chardet is very likely quite right in reporting windows-1252 aka cp1252 -- however this may be by accident as chardet sometimes seems to report that as a default when it has exhausted other possibilities.

(a) These Unicode characters cannot be decoded into latin1 or cp1252 or ascii:

Minus(−) U+2212 − or − or −
Hyphen(-) U+2010
Non-breaking hyphen  U+2011  &#8209
Figure dash(‒)  U+2012 (8210) ‒ or ‒
Horizontal bar(―) U+2015 (8213) ― or ―

What gives you the impression that they may possibly appear in the input or in the database?

(b) These Unicode characters can be decoded into cp1252 but not latin1 or ascii:

En dash(–) U+2013 (8211) –, – or –
Em dash(—) U+2014 (8212) —, — or —

These (most likely the EN DASH) are what you really need to convert to an ascii hyphen/dash. What was in the string that chardet reported as windows-1252?

(c) This can be decoded into cp1252 and latin1 but not ascii:

Soft Hyphen   U+00AD  ­

If a string contains non-ASCII characters, any attempt (using iconv or any other method) to convert it to ascii will fail, unless you use some kind of "ignore" or "replace with ?" option. Why are you trying to do that?