Issue for Type0 CMap parsing

Question

I am currently working on iOS PDF scanning using PDFKitten. I am trying to extract text for searching in PDF having Type0 font. I am not able to extract text from the PDF. Some entries in ToUnicode are missing and some are misinterpreted. Can there be issue with parsing of the CMap? If I don't have complete CMap, how should I derive it? Can I take external entries for these missing ToUnicode entries?

Thanks

Answer 1

The PDF specification offers hints on how to extract text content in section 9.10.2 Mapping Character Codes to Unicode Values:

• If the font dictionary contains a ToUnicode CMap (see 9.10.3, "ToUnicode CMaps"), use that CMap to convert the character code to Unicode.

• If the font is a simple font that uses one of the predefined encodings MacRomanEncoding, MacExpertEncoding, or WinAnsiEncoding, or that has an encoding whose Differences array includes only character names taken from the Adobe standard Latin character set and the set of named characters in the Symbol font (see Annex D):

  a) Map the character code to a character name according to Table D.1 and the font’s Differences array.

  b) Look up the character name in the Adobe Glyph List (see the Bibliography) to obtain the corresponding Unicode value.

• If the font is a composite font that uses one of the predefined CMaps listed in Table 118 (except Identity–H and Identity–V) or whose descendant CIDFont uses the Adobe-GB1, Adobe-CNS1, Adobe-Japan1, or Adobe-Korea1 character collection:

  a) Map the character code to a character identifier (CID) according to the font’s CMap.

  b) Obtain the registry and ordering of the character collection used by the font’s CMap (for example, Adobe and Japan1) from its CIDSystemInfo dictionary.

  c) Construct a second CMap name by concatenating the registry and ordering obtained in step (b) in the format registry–ordering–UCS2 (for example, Adobe–Japan1–UCS2).

  d) Obtain the CMap with the name constructed in step (c) (available from the ASN Web site; see the Bibliography).

  e) Map the CID obtained in step (a) according to the CMap obtained in step (d), producing a Unicode value.

Furthermore, as section 9.10.1 indicates,

• An ActualText entry for a structure element or marked-content sequence (see 14.9.4, "Replacement Text") may be used to specify the text content directly

According to the specification, if these methods fail to produce a Unicode value, there is no way to determine what the character code represents. This is not entirely true; e.g. embedded font programs may contain their own mappings to Unicode; but such additional sources of information are beyond the actual PDF format.

EDIT

The OP provided the file in question, iPhoneConfigurationProfileRef-2013-GM.pdf, via mail and indicated

I am getting problem for every glyph.

The issue is that ranges present in PDF are not complete and are different from adobe-identity-cmap file.

If I only use CMap embedded in PDF, I get no mapping for every character and if I use adobe one the all mappings are wrong.

As he didn't get a mapping for any glyph, let us look at the title page as an example.

The content stream contains these operation relevant for text extraction:

BT 
50 0 0 50 60 669.225 Tm 
/G1 1 Tf 
<0025> Tj 
ET 
BT 
50 0 0 50 87.6 669.225 Tm 
/G1 1 Tf 
<005100500048004b004900570054> Tj 
ET 
BT 
50 0 0 50 238 669.225 Tm 
/G1 1 Tf 
<0043> Tj 
ET 
BT 
50 0 0 50 261.45 669.225 Tm 
/G1 1 Tf 
<0056004b00510050> Tj 
ET 
BT 
50 0 0 50 355.4 669.225 Tm 
/G1 1 Tf
<0032> Tj 
ET 
BT 
50 0 0 50 380.75 669.225 Tm 
/G1 1 Tf 
<0054> Tj 
ET 
BT 
50 0 0 50 396.55 669.225 Tm 
/G1 1 Tf 
<00510048004b004e0047> Tj 
ET 
BT 50 0 0 50 60 609.225 Tm 
/G1 1 Tf 
<0034> Tj 
ET 
BT 
50 0 0 50 86.65 609.225 Tm 
/G1 1 Tf 
<00470048> Tj 
ET 
BT
50 0 0 50 125.05 609.225 Tm 
/G1 1 Tf 
<00470054> Tj 
ET 
BT 
50 0 0 50 165.45 609.225 Tm 
/G1 1 Tf 
<004700500045> Tj 
ET 
BT 
50 0 0 50 238.9 609.225 Tm 
/G1 1 Tf 
<0047> Tj 
ET

So we need to look only at the font G1 on page 1. Fortunately the font has a ToUnicode map:

/CIDInit /ProcSet findresource begin
12 dict begin
begincmap
/CIDSystemInfo <<
  /Registry (Adobe)
  /Ordering (UCS)
  /Supplement 0
>> def
/CMapName /Adobe-Identity-UCS def
/CMapType 2 def
1 begincodespacerange
<0000><FFFF>
endcodespacerange
1 beginbfchar
<000f><002d 2010>
endbfchar
15 beginbfrange
<0002><0002><0020>
<0004><000c><0022>
<000e><000e><002c>
<0010><001d><002e>
<001f><001f><003d>
<0022><0032><0040>
<0034><003d><0052>
<003f><003f><005d>
<0041><0041><005f>
<0043><005c><0061>
<005e><005e><007c>
<008a><008a><00a9>
<00a4><00a4><2014>
<00a5><00a6><201c>
<00a8><00a8><2019>
endbfrange
endcmap
CMapName currentdict /CMap defineresource pop
end
end 

Trying to apply this map one gets (based on the explicit beginbfrange...endbfrange entries):

<0025> Tj                          % "C"       = <0043>                         due to <0022><0032><0040>
<005100500048004b004900570054> Tj  % "onfigur" = <006f006e00660069006700750072> due to <0043><005c><0061>
<0043> Tj                          % "a"       = <0061>                         due to <0043><005c><0061>
<0056004b00510050> Tj              % "tion"    = <00740069006f006e>             due to <0043><005c><0061>
<0032> Tj                          % "P"       = <0050>                         due to <0022><0032><0040>
<0054> Tj                          % "r"       = <0072>                         due to <0043><005c><0061>
<00510048004b004e0047> Tj          % "ofile"   = <006f00660069006c0065>         due to <0043><005c><0061>
<0034> Tj                          % "R"       = <0052>                         due to <0034><003d><0052>
<00470048> Tj                      % "ef"      = <00650066>                     due to <0043><005c><0061>
<00470054> Tj                      % "er"      = <00650072>                     due to <0043><005c><0061>
<004700500045> Tj                  % "enc"     = <0065006e0063>                 due to <0043><005c><0061>
<0047> Tj                          % "e"       = <0065>                         due to <0043><005c><0061>

This very well matches the appearance of the page:

Answer 2

To whom it may concern, if some entries are missing from /ToUnicode CMap and the font is not referring to one of the predefined encodings/CMap(s) then Adobe Reader/Acrobat will behave like the following as I empirically observed:

• If the encoding is 1 byte code space size it will directly interpret the missing code as an Unicode code point (basically an identity cast);
• If the encoding is 2 byte code space size it will just decode the code as an invalid character.

I didn't test other combinations like variable code space size. This helped me to correctly perform text extraction in a PDF with a 1 byte encoding font that was listing only alphabetic characters in the /ToUnicode CMap, leaving out some punctuation which was encoded with the regular ASCII code.