Prelinking only has effect on relative relocations

Question

I am using cross-prelink to prelink a large C++ executables that use Qt for an embedded ARM device. Note that I am not using Yocto, but a custom distribution - so I am running prelink manually at the moment.

Looking at the output of prelink, it seems to work:

$ prelink --verbose --ld-library-path=/opt/<product>/lib:/usr/local/Qt-5.3.1/lib --root=$PRODUCT_TARGET_ROOT/<product>/rfs/ /path/to/binary
Laying out 56 libraries in virtual address space 41000000-50000000
Assigned virtual address space slots for libraries:
/lib/ld-linux.so.3                                           41000000-41027908
/opt/<product>/lib/lib<product>common.so.1                   41030000-41cf0fd0
/lib/libc.so.6                                               442b0000-443e3980
/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5                       434f0000-4380ee84
[..]
Prelinking /lib/ld-2.17.so
Prelinking /lib/libc-2.17.so
Prelinking /path/to/binary
Prelinking /<product>/lib/lib<product>common.so.1.0.0
Prelinking /usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1 
[..]

When the library gets loaded, at least libQt5Qml.so and libproductcommon.so seem to get loaded to the preferred load address set by prelink:

$ cat /proc/`pidof binary`/maps
2ab49000-2ab4a000 r--p 0001e000 07:00 9357       /roroot/lib/ld-2.17.so
2ab4a000-2ab4b000 rw-p 0001f000 07:00 9357       /roroot/lib/ld-2.17.so
2b0fd000-2b223000 r-xp 00000000 07:00 9730       /roroot/lib/libc-2.17.so
2b223000-2b22a000 ---p 00126000 07:00 9730       /roroot/lib/libc-2.17.so
2b22a000-2b22c000 r--p 00125000 07:00 9730       /roroot/lib/libc-2.17.so
2b22c000-2b22d000 rw-p 00127000 07:00 9730       /roroot/lib/libc-2.17.so
41030000-41ce7000 r-xp 00000000 07:00 9305       /roroot/<product>/lib/lib<product>common.so.1.0.0
41ce7000-41cef000 ---p 00cb7000 07:00 9305       /roroot/<product>/lib/lib<product>common.so.1.0.0
41cef000-41cf1000 rw-p 00cb7000 07:00 9305       /roroot/<product>/lib/lib<product>common.so.1.0.0
434f0000-437f8000 r-xp 00000000 07:00 1355       /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
437f8000-437ff000 ---p 00308000 07:00 1355       /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
437ff000-4380e000 rw-p 00307000 07:00 1355       /roroot/usr/local/Qt-5.3.1/lib/libQt5Qml.so.5.3.1
[..]

Now, I expected to see some reductions in the number of relocations:

$ LD_DEBUG=statistics /path/to/binary
    20453:                      number of relocations: 66379
    20453:           number of relocations from cache: 38995
    20453:             number of relative relocations: 21690

$ LD_USE_LOAD_BIAS=0 LD_DEBUG=statistics /path/to/binary
    20478:                      number of relocations: 66379
    20478:           number of relocations from cache: 38995
    20478:             number of relative relocations: 62981

This shows that only the relative relocations were reduced due to prelink, but not the normal relocations (that presumably need a symbol lookup). I am especially interested to reduce the other relocations, since those are presumably the more expensive ones.

Now my questions:

1. Is prelink even able to reduce normal relocations? An LWN article shows 0 normal relocations after prelink, so I'd assume that is possible.
2. What could I have done wrong so that non-relative relocations are not prelinked for me? Where should I start debugging?

Answer 1

OK, it turned out the problem was that some libraries were not correctly prelinked, as seen in my original question, in which e.g. libc.so wasn't loaded at the correct load address.

Seems like prelinking is a all-or-nothing approach: If one of the dependencies of an executable isn't correctly prelinked or can't be loaded at the preferred address, then neither the executable nor the libraries will be able to take advantage of prelinked symbol relocations, and only take advantage of prelinked relative relocations.

Whether a library was correctly prelinked should, in addition to the above, be checked with:

# readelf --dynamic usr/lib/someLibrary.so 
[..]
0x6ffffdf5 (GNU_PRELINKED)              2014-12-15T14:16:56
[..]

# readelf --program-headers usr/lib/someLibrary.so
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  [..]
  LOAD           0x000000 0x44bf0000 0x44bf0000 0xb56d4 0xb56d4 R E 0x8000
  [..]

The address outputted by prelink --verbose, readelf --program-headers and cat /proc/PID/maps need to match.

My mistake was that I didn't check readelf - if I had done so, I would have realized that some libraries on the target device were not prelinked, because an error in the buildsystem caused the prelinked versions to be overwritten with the non-prelinked ones...

After fixing my buildsystem problem, the normal relocations indeed went down to 0:

# LD_DEBUG=statistics /path/to/binary
  5089:                      number of relocations: 0
  5089:           number of relocations from cache: 19477
  5089:             number of relative relocations: 0

Answer 2

It seems some linked libraries are not prelinked, with prelink info outdated or with collisions in the assigned addresses.

Either that or you might have bad luck and suffer something like that http://lwn.net/Articles/341313/:

About 10% to 50% of the time on i686, this benefit of prelink is trashed by the randomization of the placement of [vdso], also known as linux-gate.so. If the page that the kernel chooses for [vdso] overlaps any pre-linked needed shared library, then ld-linux cannot avoid processing the relocations for that library. Often the cost snowballs as libraries that do not get their pre-linked pages are moved so that they interfere with subsequent libraries. [On x86_64 the vdso is at a special fixed address that cannot conflict.]

Try this example from https://bugzilla.redhat.com/show_bug.cgi?id=162797

    for i in 0 1 2 3 4 5 6 7 8 9; do
      for j in 0 1 2 3 4 5 6 7 8 9; do
        for k in 0 1 2 3 4 5 6 7 8 9; do
          ldd /bin/cat
        done
      done 
    done  |  grep libc  |  sort  |  uniq -c

For current Fedora 11 on i686, I see a conflict about 10% of the time, involving only ld-linux, libc, and [vdso]. This means that glibc must be dynamically relocated about 10% of the time anyway, even though glibc has been pre-linked, and even though /bin/cat is near minimal in its use of shared libraries. When a GNOME app uses 50 or more pre-linked shared libs, as claimed in another thread on this subject, then runtime conflict and expense are even more likely.

Note that arm does not have a [vdso] but has a code page full of utility functions documented here https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt they are located inside the [vectors] ffff0000-ffff1000.

I would try running the prelink stage directly on the platform, although it seems that it's a embedded one with a read-only fs, but I think it's worth a try.

Note that these relocations only affect startup time of the program.