Executing user-space programs from a different Linux distro
While trying to use executables from one ARM-based distribution to another, it failed to run, even before trying to load any libraries. The ARM architectures were compatible (armhf in both cases) so it wasn’t like I was trying to run an Intel binary on an ARM. I could always cross-compile from sources, but copying binaries is much easier…
I’ll demonstrate this issue with the “ls” program. Of course I tried to adopt something more worthy.
It was just like (where the current directory’s “ls” is the binary belonging to the other distro)
# ./ls -bash: ./ls: No such file or directory
or sometimes (depends on the distribution) it says
$ ./ls -sh: ./ls: not found
or when attempting to run with bash:
$ bash ./ls ./ls: ./ls: cannot execute binary file
Attempting to set LD_DEBUG=all was pointless, because the error was earlier on. Strace gave an idea:
$ strace ./ls
execve("./ls", ["./ls"], [/* 13 vars */]) = -1 ENOENT (No such file or directory)
dup(2) = 3
fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR)
fstat64(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aac9000
_llseek(3, 0, 0x7efca940, SEEK_CUR) = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: No such file or di"..., 40strace: exec: No such file or directory
) = 40
close(3) = 0
munmap(0x2aac9000, 4096) = 0
exit_group(1) = ?
So execve() returns ENOENT even though the file exists. Which means, in this case, that the file is there but the kernel refuses to run it.
The reason
The crucial difference between the alien “ls” and the native one, is the where they expect to find their loader:
$ readelf -l /bin/ls Elf file type is EXEC (Executable file) Entry point 0xcb84 There are 7 program headers, starting at offset 52 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align EXIDX 0x093b4c 0x0009bb4c 0x0009bb4c 0x00110 0x00110 R 0x4 PHDR 0x000034 0x00008034 0x00008034 0x000e0 0x000e0 R E 0x4 INTERP 0x000114 0x00008114 0x00008114 0x00013 0x00013 R 0x1 [Requesting program interpreter: /lib/ld-linux.so.3] LOAD 0x000000 0x00008000 0x00008000 0x93c60 0x93c60 R E 0x8000 LOAD 0x094000 0x000a4000 0x000a4000 0x007bd 0x02a88 RW 0x8000 DYNAMIC 0x09400c 0x000a400c 0x000a400c 0x000f0 0x000f0 RW 0x4 GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4 Section to Segment mapping: Segment Sections... 00 .ARM.exidx 01 02 .interp 03 .interp .hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.extab .ARM.exidx .eh_frame 04 .init_array .fini_array .jcr .dynamic .got .data .bss 05 .dynamic 06 $ readelf -l ./ls Elf file type is EXEC (Executable file) Entry point 0xb6d9 There are 9 program headers, starting at offset 52 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align EXIDX 0x00fce8 0x00017ce8 0x00017ce8 0x00030 0x00030 R 0x4 PHDR 0x000034 0x00008034 0x00008034 0x00120 0x00120 R E 0x4 INTERP 0x000154 0x00008154 0x00008154 0x00027 0x00027 R 0x1 [Requesting program interpreter: /lib/arm-linux-gnueabihf/ld-linux.so.3] LOAD 0x000000 0x00008000 0x00008000 0x0fd1c 0x0fd1c R E 0x8000 LOAD 0x00fee4 0x0001fee4 0x0001fee4 0x003e4 0x01050 RW 0x8000 DYNAMIC 0x00fef0 0x0001fef0 0x0001fef0 0x00110 0x00110 RW 0x4 NOTE 0x00017c 0x0000817c 0x0000817c 0x00044 0x00044 R 0x4 GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4 GNU_RELRO 0x00fee4 0x0001fee4 0x0001fee4 0x0011c 0x0011c R 0x1 Section to Segment mapping: Segment Sections... 00 .ARM.exidx 01 02 .interp 03 .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.exidx .eh_frame 04 .init_array .fini_array .jcr .dynamic .got .data .bss 05 .dynamic 06 .note.ABI-tag .note.gnu.build-id 07 08 .init_array .fini_array .jcr .dynamic
Aha! When the native “ls” is executed, the kernel loads /lib/ld-linux.so.3 which in turn executes the required executable. When the alien “ls” was attempted, the kernel went for /lib/arm-linux-gnueabihf/ld-linux.so.3, couldn’t find it and returned “no such file”. It actually means that it didn’t find the interpreter binary (i.e. the glibc dynamic library loader).
The Solution
Create a symlink from where the executable expects the loader to where it actually is. In this case
# mkdir /lib/arm-linux-gnueabihf # cd /lib/arm-linux-gnueabihf # ln -s /lib/ld-linux.so.3
It’s of course quite likely that some library binaries will need to be copied along with the executable. LD_DEBUG or ldd may be helpful here, as well as “readelf -d” if there’s no ldd.
Changing the dynamic linker when compiling
Sometimes it’s possible to go the other way around: Tell gcc to pick a certain dynamic linker.
But first, to see which loader a program compiled with gcc will expect, add the -v flag in the compilation command, e.g.
$ gcc -v -O3 -Wall tryexec.c -o tryexec
and look for the -dynamic-linker flag in COLLECT_GCC_OPTIONS (could be, for example, /lib64/ld-linux-x86-64.so.2).
To change the choice of linker, pass an argument to the linker through gcc with the -Wl flag:
$ gcc -O3 -Wl,-I/lib/ld-linux.so.3 -Wall tryexec.c -o tryexec
What comes after the comma of the -Wl flag goes to the linker, so -Wl,-I/lib/ld-linux.so.3 passes “-I/lib/ld-linux.so.3″ to ld, which does the job.
Those using Eclipse (Xilinx SDK included) can add the flag in the project C/C++ Build Settings > Tool Settings > ARM Linux gcc linker > Miscellaneous > Linker Flags (write e.g. “-Wl,-I/lib/myloader.so”, without the quotes, in the text box).