Linux on Microblaze HOWTO (part II)

This is part II of my HOWTO on running Linux on Microblaze. The outline is as follows:

• Part I: Introduction and setting up the Microblaze processor
• Part II: Compiling the kernel (this page)
• Part III: Preparing for boot and booting
• Part IV: Compiling user space applications

Kernel compilation in general

Compiling a Linux kernel traditionally consists of the following steps (some of which are elaborated further below):

• Obtaining a kernel source tree.
• Configure the kernel. Which all in all means to set up a file named “.config” in the kernel source’s root directory.
• Compile actual kernel, ending up with an executable image.
• Compile the post-boot loadable kernel modules.
• Put everything in its place, set up the bootloader
• Pray and boot

When compiling for Microblaze, the process is somewhat different:

• Cross compilation: The compiled binaries run on a processor different from the one doing the compilation.
• Kernel modules are most likely not used at all. They are a bit pointless when the hardware is known in advance, and also add some complexity in setting up the entire system for boot. Besides, modprobe on a Microblaze can take forever.
• The hardware configuration is custom made, and the kernel needs to be informed about it (through the Device Tree Structure)

Downloading kernel sources

Note that all kernels compile for all target architectures. If you download a kernel from Xilinx’ repository, it may have the parts relevant to Xilinx slightly more updated. The emphasis is on “may”.
The “vanilla” kernel (maintained by Linus Torvalds) can be downloaded from the main kernel archive or one of its mirrors. Several other flavors float around, including Xilinx own git

git clone git://git.xilinx.com/linux-2.6-xlnx.git

or Petalogix’ git (after all, they do a lot of maintenance on the Xilinx devices):

git clone git://developer.petalogix.com/linux-2.6-microblaze.git

The question is always which kernel is best. The answer is that it’s a bit of a gamble. It’s usually almost exactly the same piece of software, with git version having the latest changes. That means the latest bug fixes, new drivers, but also the latest, undocumented and undiscovered bugs. Vanilla kernels tend to be more conservative, but the only rule is that there are no rules. So in short, toss a coin and pick one.

Personally, I compiled the kernel which happened to be on my hard disk for other purposes.

Cross compilers

The good news is that there’s no need to compile the GNU tools. As a matter of fact, this part turned out to be surprisingly painless. The cross compiler and binutils binaries + initramfs images can be downloaded with

$ git clone git://git.xilinx.com/xldk/microblaze_v1.0_le.git
$ git clone git://git.xilinx.com/xldk/microblaze_v1.0.git

Choose one, depending on whether you prefer little endian or big endian for your processor. I picked little endian, but there’s one initramfs in the big endian bundle which isn’t there for the little endian set (which only has the “minimal” image).

One of the files fetched by git is microblazeel-unknown-linux-gnu.tar.gz (gzipped tarball) for the little endian version and mb_gnu_tools_bin.tar.bz (bzipped tarball) for big endian. I’ll leave the latter, because I didn’t use it.

There’s no need to install anything, and no need to be root (actually, doing this as root is pretty unwise). Just untar the tarball of your choice in any directory. Tar generates several subdirectories, but we’re after the cross compilers. Or more precisely, to make the kernel build system use them. This boils down to this:

export CROSS_COMPILE=/home/myhomedir/untarred-to/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-

First of all, note the dash at the end of the statement. The whole string is a prefix for all compilation commands made by the kernel build system. It is often recommended to set the path to where the compilers are, and then set CROSS_COMPILE to a shorter prefix. I don’t see the point in polluting the overall path. The build environment has no problem with the statement above.

It has also crossed my mind to use the mb-gcc and friends, which are part of the SDK. But that may require another paid-for license, in case different people do the FPGA and software (which usually is the case).

And to wrap this up: If I’ll ever need to build a cross compiler from scratch, I would start with looking at Buildroot (and another page about it) or following this guide (I haven’t tried either, though).

Kernel configuration

Setting this up correctly is a tedious process, and even the most seasoned kernel hackers may not get it right on the first go. If it’s your first time, prepare to spend quite a few hours on this. The less experienced you are with Linux in general, the more time will you need to spend to make an educated guess about your need for each feature offered.

You can try to use my configuration file, or at least start off with it. It was made for against a 2.6.38 kernel, and booted well as shown in part III. Copy the file as .config on the kernel source’s root, and start with oldconfig.

The commands involved are basically (all “make” commands issues at the kernel source’s top directory):

• Clean up everything, including the .config file if present. This is not necessary if you just uncompressed your kernel. It’s actually rarely necessary at all: “make ARCH=microblaze mrproper”. This will delete .config! (I know I just said it).
• Adopt an existing .config file: “make ARCH=microblaze oldconfig”. This is useful in particular when switching to another kernel version or flavor. Only questions about new features are asked. If you downloaded my configuration file, I would suggest not to turn on options that are offered while running oldconfig, unless they are clearly Xilinx related.
• Configure the kernel: “make ARCH=microblaze xconfig”, “make ARCH=microblaze gconfig” or “make ARCH=microblaze menuconfig” (pick one). These applications present the kernel options in a fairly user-friendly manner, and eventually save the result to .config. I recommend xconfig, because it’s graphic and has a search feature, which turns out very useful.

When targeting an embedded platform, the strategy is to enable whatever is necessary in the kernel itself, and not count on kernel modules. A second issue is to eliminate anything unnecessary from the kernel. This is not just a matter of the kernel image’s size and speed, but enabling components which have nothing to do there can cause the kernel compilation to fail, and even worse, the kernel to crash at boot. Each architecture maintains a set of #include headers, and some kernel components may assume certain things that these architecture-dependent parts haven’t caught up with. So the rule that is whatever hasn’t been tested, won’t necessarily work. Enabling an esoteric keyboard driver on a Microblaze processor may very well fail the boot, simply because nobody cares.

In particular, you most likely want to follow these:

• Under Platform Options, set CONFIG_KERNEL_BASE_ADDR to where your DDR RAM begins (0xC0000000 on my processor), the targeted FPGA family as well as the other parameters (USE_*). The USE_* parameters’ correct values can be found in the .dts file. Just copy the values of the processor elements with the same names.
• Also set
  CONFIG_SERIAL_UARTLITE=y
  CONFIG_SERIAL_UARTLITE_CONSOLE=y
  CONFIG_SERIAL_CORE=y
  CONFIG_SERIAL_CORE_CONSOLE=y
• Since we’re not going to use any boot loader, the kernel command line needs to be compiled into the kernel itself: Enable CMDLINE_BOOL (default bootloader kernel argument) and set it to something useful. As for the console, set it to console=ttyUL0, or nothing goes to console after the two first lines sent to console from early_printk_console (CONFIG_CMDLINE_FORCE may be necessary as well. It doesn’t hurt in the absence of a boot loader anyhow)
• Enable CONFIG_MSDOS_FS and CONFIG_VFAT_FS in kernel (not module), so that the SystemACE can be read.
• Enable CONFIG_XILINX_SYSACE
• Enable CONFIG_XILINX_EMACLITE and CONFIG_FB_XILINX
• Disable the FTRACE config option (under kernel hacking, compilation fails) instead of using patch.

And for your own sake, make a copy of the .config file every now and then as you work on it. It’s very easy to delete it by mistake or to mess it up in general.

Setting the Linux boot parameters correctly is very important, because if they’re wrong, kernel recompilation is they only way to fix it in the absence of a boot loader. I’ve chosen to mount the root directory from the network, but note that /dev/sxa is the Compact flash itself (with /dev/sxa1 is the first partition, for example). So it’s fairly simple to add a partition to the flash device, and put a regular root filesystem there. Maybe I’ll do that myself and update this post.

Anyhow, my choice for the Linux boot parameters was

console=ttyUL0 ip=::::::dhcp rootfstype=nfs root=/dev/nfs rw nfsroot=10.11.12.13:/shared/nfsroot,tcp

where “/shared/nfsroot” is the shared NFS directory on the server with IP 10.11.12.13. This command is suitable for getting the root from the network, which is very convenient for development. This setting requires a DHCP server on the LAN. In case you don’t want to configure a DHCP server, use the ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:::off format instead. Documentation/filesystems/nfs/nfsroot.txt in the kernel sources has more about booting from NFS. I’ve also written a post about booting a diskless PC from network, but it’s a bit of an overkill.

In case you’re interested in how the whole configuration thing comes together, let’s take CONFIG_EARLY_PRINTK for example. In arch/microblaze/kernel/Makefile, one of the lines says:

obj-$(CONFIG_EARLY_PRINTK)    += early_printk.o

On the other hand, in the config file it can say

CONFIG_EARLY_PRINTK=y

So when the Makefile is executed, the target early_prink.o is added to either obj-y, obj-m or obj-n. obj-y is the list of objects to be inserted into the kernel, obj-m is the list of modules, and obj- is the junk list. The configuration rules are given in the Kbuild files, next to the Makefiles.

A small Makefile fix

As of 2.6.38, there is a small error in the arch/microblaze/boot/Makefile, which makes the build system always attempt making an U-Boot image, which is not necessary in our case. This may result in an error message (saying “mkimage” wasn’t found), when everything is actually OK. So in the part saying

$(obj)/simpleImage.%: vmlinux FORCE
 $(call if_changed,cp,.unstrip)
 $(call if_changed,objcopy)
 $(call if_changed,uimage)
 $(call if_changed,strip)
 @echo 'Kernel: $@ is ready' ' (#'`cat .version`')'

remove or comment out the line saying “$(call if_changed,uimage)”.

Compiling the kernel

Before starting: You didn’t forget to set CROSS_COMPILE and copy the updated xilinx.dts file to its place… right?

I prefer cleaning up before compiling:

make ARCH=microblaze clean
rm arch/microblaze/boot/simpleImage.*

This is a good time to ask why the image file isn’t cleaned by “make clean”. To be fixed, I suppose.

And then, the compilation is just

make -j 8 ARCH=microblaze simpleImage.xilinx

Note that the “.xilinx” suffix corresponds to the xilinx.dts file in the arch/microblaze/boot/dts/ directory. If another .dts file should be made effective, change the suffix.

The “-j 8″ means that 8 compilation processes run in parallel, which is suitable for a quad processor with hyperthreading. Skip this option or use another number, depending on your computer, your spare time and your need to see the logic of the events.

The basic UNIX rule is that everything went fine unless an error message appeared. A more explicit confirmation is that it said

OBJCOPY arch/microblaze/boot/simpleImage.xilinx

somewhere close to the end, and that the arch/microblaze/boot/simpleImage.xilinx is indeed there, and has a date stamp that makes sense.

If and when you get errors, well, there’s no simple  recipe to solve that. The easiest way is to eliminate the need to compile that certain file by changing the kernel configuration, if the functionality is indeed unnecessary. Otherwise your best friends are Google and your brain, not necessarily in that order.

As for the Device Tree, it was compiled into a .dtb file (the Device Tree binary blob), which can be found in the same directory as the just generated kernel image. The Device Tree Compiler (dtc) comes with the kernel sources, and can be found in scripts/dtc.

And just to wrap this up: If you insist on seeing all the commands issued instead of the otherwise laconic output, there the KBUILD_VERBOSE flag. For example,

make ARCH=microblaze KBUILD_VERBOSE=1 clean

With a compiled kernel image at hand (which already has the Device Tree built-in), all that’s left is to set up the Compact Flash and boot. Go to part III of this HOWTO.

A few other make statements

For completeness:

• Clean up any compiled binaries: Recommended after a change in .config: “make ARCH=microblaze clean”
• Generate loadable modules: “make ARCH=microblaze modules”. Not necessary if everything needed is compiled into the kernel.
• And then gather the modules in a neat directory (making sure you don’t have a /lib/modules directory with the same version number): “make ARCH=microblaze modules_install”. This will write to /lib/modules on the local machine, so if you happen to compile exactly the same kernel version for your own PC and the embedded target, the kernel modules the PC relies on will be overwritten.