How to builld OS/ABI for SystemV using crosstool-ng

Question

I am trying to build a kernel module for my DNS-320L NAS.

I have built crosstool-ng in a Debian lenny chroot environment but after I have compiled my kernel module and try and install it, I get:

insmod: error inserting 'kernel/net/ip4/ipip.ko': -1 Invalid module format

When I examine a working kernel module with readelf, I get:

~/ct-ng-build$ readelf -h ~/ct-ng-build/kernel/orig/ipip.ko 
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          6696 (bytes into file)
  Flags:                             0x5000000, Version5 EABI
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         51

But my module has a few differences:

~/ct-ng-build$ readelf -h ~/ct-ng-build/kernel/modules/lib/modules/2.6.31.8/kernel/net/ipv4/ipip.ko 
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 61 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            ARM
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          6280 (bytes into file)
  Flags:                             0x600, GNU EABI, software FP, VFP
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         23
  Section header string table index: 20

My cross_compile environment is:

$ echo $CROSS_COMPILE 
arm-none-eabi-


make CROSS_COMPILE=${CROSS_COMPILE} INSTALL_PATH=~/ct-ng-build/kernel/install INSTALL_MOD_PATH=~/ct-ng-build/kernel/modules INSTALL_FW_PATH=~/ct-ng-build/kernel/firmware

My gcc is compiled:

$  ${CROSS_COMPILE}gcc -v
Using built-in specs.
Target: arm-none-eabi
Configured with: ~/ct-ng-build/targets/src/gcc-4.3.2/configure --build=x86_64-build_unknown-linux-gnu --host=x86_64-build_unknown-linux-gnu --target=arm-none-eabi --prefix=~/x-tools/arm-none-eabi --with-local-prefix=~/x-tools/arm-none-eabi/arm-none-eabi//sys-root --disable-multilib --disable-libmudflap --with-sysroot=~/x-tools/arm-none-eabi/arm-none-eabi//sys-root --with-newlib --enable-threads=no --disable-shared --with-pkgversion=crosstool-NG-1.9.0 --with-arch=armv5te --with-tune=arm926ej-s --disable-__cxa_atexit --with-gmp=~/ct-ng-build/targets/arm-none-eabi/build/static --with-mpfr=~/ct-ng-build/targets/arm-none-eabi/build/static --enable-target-optspace --disable-nls --enable-symvers=gnu --enable-languages=c,c++
Thread model: single
gcc version 4.3.2 (crosstool-NG-1.9.0)

Any ideas how to make the OS/ABI to "UNIX - System V" and the Flags to "0x5000000, Version5 EABI" instead of "Arm" and "0x600, GNU EABI, software FP, VFP" ?

Or is there another problem?

Thanks!

--- EDIT ---- I am specifically trying to target my existing NAS OS:

Kernel  2.6.31.8 #1 armv5tel
C library   gcc-4.3-mt-1.44.0 

# /lib/libc-2.8.so 
GNU C Library stable release version 2.8, by Roland McGrath et al.
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.3.2.

I have only been able to build a crosstool-ng environment so far using crosstool-ng-1.9.0, but it produced the binaries above.

I tried to set -mabi=apcs-gnu as per 3.17.2 ARM Options but the C library would not compile with strange "missing headers" errors (I may try again - possibly something with my chroot environment).

Answer 1

Build a new toolchain using crosstool-NG, but configure it for OABI rather than EABI.
See Configuring crosstool-NG.

All debian releases (for ARM) upto and including Lenny were OABI.

Ref: https://wiki.embeddedarm.com/wiki/EABI_vs_OABI

Even though your toolchain prefix is arm-none-eabi-, the binary it has produced looks like OABI.
I have OABI binaries that produce identical readelf output, and EABI binaries that have identical readelf output ("UNIX - System V") to what you want.

This question describes a similar but reversed situation to yours, i.e. his toolchain is generating EABI binaries but he needs OABI.

Apparently your toolchain was built to produce OABI but a misleading prefix was used.
You need to build a new toolchain using crosstool-NG, but configure it for EABI rather than OABI.
Newer versions of crosstool-NG actually make configuration for OABI difficult by forcing the selection EABI unless Use obsolete features is enabled.

---

ADDENDUM

You say "but configure it for OABI rather than EABI.", but how?

Using a crosstool-NG v1.18 installation, I can specify & build
* Linux kernel version 2.6.31.14,
* gcc version 4.3.2,
* binutils version 2.18a 2.19.1a,
* glibc version 2.8.

In order to unselect Target options ---> Use EABI (unselecting means use OABI),
I first have to select Paths and misc options ---> Use obsolete features.

In this version of crosstool-NG, Use EABI is automatically selected.

---

The description for the Use EABI menu item has:

Set up the toolchain so that it generates EABI-compliant binaries.

If you say 'n' here, then the toolchain will generate OABI binaries.
OABI has long been deprecated, and is now considered legacy.

Since OABI is considered deprecated in this version of crosstool-NG, the config symbol ARCH_ARM_EABI_FORCE is active unless Use obsolete features is specified.

Answer 2

I'm extracting some final comments here:

As @sawdust recommends, I am using EABI in the following CT-NG environment:

crosstool-NG version: 1.9.0
gcc 4.3.2
binutils 2.19.1
libc 2.8
gmp 4.3.2
mpfr 2.4.2

In order to get the GCC compiler to compile, I had to do a couple of hacks! Namely:

• remove res_hconf.c from targets/src/glibc-2.8/nscd (once, after the sources have been extracted)
• remove res_hconf from the list of objects in the nscd/Makefile
• create a soft link to stubs-32.h from stubs-.h in ~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root/usr/include/gnu (after the build had started).

I just had to download the correct Linux kernel and install it. The missing command was:

make ARCH=arm kirkwood_defconfig

Thereafter I just had to use make ARCH=arm menuconfig to customise things.

The gcc -v says:

Using built-in specs.
Target: arm-unknown-linux-gnueabi
Configured with: ~/ct-ng-build/targets/src/gcc-4.3.2/configure --build=x86_64-build_unknown-linux-gnu --host=x86_64-build_unknown-linux-gnu --target=arm-unknown-linux-gnueabi --prefix=~/x-tools/arm-unknown-linux-gnueabi --with-sysroot=~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root --enable-languages=c,c++ --disable-multilib --with-arch=armv5te --with-tune=arm926ej-s --with-float=soft --disable-shared --with-pkgversion=crosstool-NG-1.9.0 --disable-sjlj-exceptions --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --with-gmp=~/ct-ng-build/targets/arm-unknown-linux-gnueabi/build/static --with-mpfr=~/ct-ng-build/targets/arm-unknown-linux-gnueabi/build/static --enable-threads=posix --enable-target-optspace --with-local-prefix=~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root --disable-nls --enable-symvers=gnu --enable-c99 --enable-long-long
Thread model: posix
gcc version 4.3.2 (crosstool-NG-1.9.0) 

Finally, to generate SYSV modules, I had to hack the kernel arch/arm/Makefile to remove the -mabi=aapcs-linux setting from the EABI section.

Now the error is:

"modprobe: can't load module ntfs (kernel/fs/ntfs/ntfs.ko): unknown symbol in module, or unknown parameter

Which is for another question :D

I still can't build a working kernel module (maybe because of the hacks!) but it might also be because I don't have a compatible kernel .config for the NAS.

The final changes were:

• Not use unwind
• Disable some kernel debugging and tracing
• use SLAB instead of SLUB

The changes were selected by tracking the dmsg messages.

The resulting compiled kernel modules can now be loaded, if they are compatible with the compiled kernel.

(Note of warning: I got into a little trouble when I started adding iptables modules when I got locked out of my network connection! Be careful! Fortunately the modules are not reloaded at reboot. )