Scope
Even though Xilinx supplies a cute wizard for creating peripherals in its EDK (version 13.2 in my case), it’s just enough to work as a demo. For a real-life case there’s no escape from getting down to the system’s guts. As it turns out, things are pretty well organized under EDK’s hood, which makes the attempt to cover it all up with a wizard even more questionable.
This post is a jot-down of the technicalities behind designing a minimal bare-boned peripheral and its Linux driver. With “bare-boned” I mean that it has the absolutely simplest bus interface (AXI4 Lite without even decoding the addresses), and that it’s in pure Verilog. No external IP is used.
This peripheral connects to the SP605′s four LEDs. Any write to its address region updates the LED’s state according to the written value’s four LSBs. Reading back from any of its covered addresses gives the four LSBs back. That’s all.
Sources of information
This post assumes that you’re familiar with running Linux on Microblaze. I have a pretty extensive tutorial on the subject for reference.
These are worth to look at:
- The official cores’ sources (in VHDL) can be found at ISE_DS\EDK\hw\XilinxProcessorIPLib\pcores (path from where ISE is installed). It’s interesting in particular to look at axi_lite_ipif_v1_00_a.
- The AMBA AXI protocol specification, downloaded free from ARM’s site.
- Platform Specification Format Reference Manual (UG642, psf_rm.pdf): Describes the file formats in detail. Necessary when editing the files.
- EDK Concepts, Tools, and Techniques (UG683, edk_ctt.pdf) : The chapter about Creating Your Own Intellectual Property is somewhat helpful to try out the Wizard.
Understanding the process
It looks like the missing link in Xilinx’ documentation is to explain how the whole machinery works with regard to adopting a custom made peripheral. I’ll try to fill in that gap now.
Generally speaking, the minimal core consists of the following files, which should be in a dedicated directory under the “pcores” directory, which is under the EDK project’s top directory:
- data/minimal_v2_1_0.mpd: This file is what EDK looks at when an IP is added to a project. It contains all the information used directly by the EDK. The peripheral’s GUI is set up according to this information, and it’s also used when the EDK generates wrappers and connections for it. Its format is well documented, but it looks like it’s easier to just copy snippets from existing core’s MPD files. It’s also possible to generate this file automatically with PsfUtility from the toplevel source file, but it’s not clear if it’s worth the effort to learn yet another tool.
- data/minimal_v2_1_0.pao: This file supplies EDK with a list of HDL files which need to be synthesized to create the peripheral. It also sets the order of synthesis.
- hdl/verilog/minimal.v: The Verilog file constituting the peripheral. Of course there may be several files, which need to be mentioned in the PAO file.
- Note that “black box” modules (presynthesized netlists) are listed in BBD files, which are not necessary in this example. When used, the MPD file is set to reflect this.
The file names above relate to a peripheral called “minimal”. They change according to the project’s setting and version numbers.
All in all, the flow is pretty simple: Only the MPD file is considered by EDK, and only at platform netlist generation are the HDL files synthesized according to the PAO file. The instantiation and connection depend on the settings within the EDK (actually, the MHS file).
It’s easiest to create just any peripheral with the wizard, see what they do, and then modify the files.
Going from Wizard’s output to minimal peripheral
This is a short outline of the stages. The result is given in the next section.
- Edit the data/*.pao file: Remove all files and insert the single Verilog file, changing the type to verilog.
- In the data/*.mpd file, change OPTION HDL = VHDL to VERILOG too. Also add, under ##ports, PORT minimal_leds = “”, DIR = O, VEC = [3:0] (so that the I/O port is known. Note the =”" part).
- Remove data/*.prj file so no unnecessary files are included (even though this file seems to be ignored).
- Roughly translate the VHDL file to Verilog. Make it Verilog parameters instead of VHDL generics.
- Rename/remove the devl directory, since its information is not in sync with the new situation, and Custom IP Wizard can’t do any good at this point.
- And finally, in EDK, Project > Rescan User Repositories
- Remove the LED_4bits core from the project, choosing “Delete instance and any connections to internal nets”. This will keep the net names used for connecting to the LEDs, and make them available for connection to the new peripheral. Otherwise, the external net names need to be set, and the system.ucf given at the “project” tab updated to reflect the new nets.
- Add the minimal core to the project, and connect the just released LEDs_4Bits_TRI_O to its minimal_leds port.
- Create bitfile
The synthesis of the peripheral’s HDL takes place during the “create netlist” flow (which is, of course, part of generating bitfile). For example, the synthesis of an instance named minimal_0 will appear as follows in the console
INSTANCE:minimal_0 - C:\tryperipheral\system.mhs line 424 - Running XST
synthesis
PMSPEC -- Overriding Xilinx file
<C:/ise13_2/ISE_DS/EDK/spartan6/data/spartan6.acd> with local file
<C:/ise13_2/ISE_DS/ISE/spartan6/data/spartan6.acd>
And if there are errors in the HDL, they will show up at this point.
Sample files
These are the files used for the minimal peripheral. They are a sloppy adoption of the files generated by the Custom IP Wizard, so they’re very likely to contain unnecessary declarations.
First, the Verilog file:
module minimal #(
parameter C_S_AXI_DATA_WIDTH = 32,
parameter C_S_AXI_ADDR_WIDTH = 32,
parameter C_S_AXI_MIN_SIZE = 'h000001FF,
parameter C_USE_WSTRB = 0,
parameter C_DPHASE_TIMEOUT = 8,
parameter C_BASEADDR = 'hFFFFFFFF,
parameter C_HIGHADDR = 'h00000000,
parameter C_FAMILY = "spartan6",
parameter C_NUM_REG = 1,
parameter C_NUM_MEM = 1,
parameter C_SLV_AWIDTH = 32,
parameter C_SLV_DWIDTH = 32
)
(
input S_AXI_ACLK,
input S_AXI_ARESETN,
input [(C_S_AXI_ADDR_WIDTH-1):0] S_AXI_AWADDR,
input S_AXI_AWVALID,
input [(C_S_AXI_DATA_WIDTH-1):0] S_AXI_WDATA,
input [((C_S_AXI_DATA_WIDTH/8)-1):0] S_AXI_WSTRB,
input S_AXI_WVALID,
input S_AXI_BREADY,
input [(C_S_AXI_ADDR_WIDTH-1):0] S_AXI_ARADDR,
input S_AXI_ARVALID,
input S_AXI_RREADY,
output S_AXI_ARREADY,
output [(C_S_AXI_DATA_WIDTH-1):0] S_AXI_RDATA,
output [1:0] S_AXI_RRESP,
output S_AXI_RVALID,
output S_AXI_WREADY,
output [1:0] S_AXI_BRESP,
output reg S_AXI_BVALID,
output S_AXI_AWREADY,
output reg [3:0] minimal_leds
);
assign S_AXI_RDATA = minimal_leds;
assign S_AXI_RRESP = 0; // OKAY on AXI4
assign S_AXI_ARREADY = 1; // Always ready for read address
assign S_AXI_AWREADY = 1; // Always ready for write address
assign S_AXI_RVALID = 1; // Read data always valid (ILLEGAL)
assign S_AXI_WREADY = 1; // Always ready to write
assign S_AXI_BRESP = 0; // OKAY on AXI4
// This will not work OK if several "bursts" are sent with no BVALIDs
// inbetween. Not an expected scenario.
always @(posedge S_AXI_ACLK)
if (S_AXI_WVALID)
begin
S_AXI_BVALID <= 1;
minimal_leds <= S_AXI_WDATA;
end
else if (S_AXI_BREADY && S_AXI_BVALID) // Active BRESP cycle
S_AXI_BVALID <= 0;
endmodule
Most of the parameters at the top can be removed, I believe. It appears like they are necessary only when creating the MPD file with PsfUtility.
All ports, except minimal_leds are standard AXI4 lite ports. The implementation of the interface isn’t example for anything except a quick and dirty peripheral which responds to bus requests. The only thing it does actively is to update minimal_leds when necessary, and toggle the AXI_BVALID, so that only one burst response is sent for each write cycle (which is always one clock long in AXI4 lite). It’s OK not to decode the address, since it’s the interconnect’s job to make sure each peripheral gets only what it directed to it.
Holding S_AXI_RVALID high all the time violates the AXI4 spec, since it’s required to be asserted only after ARVALID and ARREADY. But the interconnect tolerated this anyhow.
Now to minimal_v2_1_0.mpd:
BEGIN minimal
## Peripheral Options
OPTION IPTYPE = PERIPHERAL
OPTION IMP_NETLIST = TRUE
OPTION HDL = VERILOG
OPTION IP_GROUP = MICROBLAZE:USER
OPTION DESC = MINIMAL
OPTION LONG_DESC = A minimal peripheral to start off with
OPTION ARCH_SUPPORT_MAP = (others=DEVELOPMENT)
## Bus Interfaces
BUS_INTERFACE BUS = S_AXI, BUS_STD = AXI, BUS_TYPE = SLAVE
## Generics for VHDL or Parameters for Verilog
PARAMETER C_S_AXI_DATA_WIDTH = 32, DT = INTEGER, BUS = S_AXI, ASSIGNMENT = CONSTANT
PARAMETER C_S_AXI_ADDR_WIDTH = 32, DT = INTEGER, BUS = S_AXI, ASSIGNMENT = CONSTANT
PARAMETER C_S_AXI_MIN_SIZE = 0x000001ff, DT = std_logic_vector, BUS = S_AXI
PARAMETER C_USE_WSTRB = 0, DT = INTEGER
PARAMETER C_DPHASE_TIMEOUT = 8, DT = INTEGER
PARAMETER C_BASEADDR = 0xffffffff, DT = std_logic_vector, MIN_SIZE = 0x0, PAIR = C_HIGHADDR, ADDRESS = BASE, BUS = S_AXI
PARAMETER C_HIGHADDR = 0x00000000, DT = std_logic_vector, PAIR = C_BASEADDR, ADDRESS = HIGH, BUS = S_AXI
PARAMETER C_FAMILY = virtex6, DT = STRING
PARAMETER C_NUM_REG = 1, DT = INTEGER
PARAMETER C_NUM_MEM = 1, DT = INTEGER
PARAMETER C_SLV_AWIDTH = 32, DT = INTEGER
PARAMETER C_SLV_DWIDTH = 32, DT = INTEGER
PARAMETER C_S_AXI_PROTOCOL = AXI4LITE, TYPE = NON_HDL, ASSIGNMENT = CONSTANT, DT = STRING, BUS = S_AXI
## Ports
PORT S_AXI_ACLK = "", DIR = I, SIGIS = CLK, BUS = S_AXI
PORT S_AXI_ARESETN = ARESETN, DIR = I, SIGIS = RST, BUS = S_AXI
PORT S_AXI_AWADDR = AWADDR, DIR = I, VEC = [(C_S_AXI_ADDR_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_AWVALID = AWVALID, DIR = I, BUS = S_AXI
PORT S_AXI_WDATA = WDATA, DIR = I, VEC = [(C_S_AXI_DATA_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_WSTRB = WSTRB, DIR = I, VEC = [((C_S_AXI_DATA_WIDTH/8)-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_WVALID = WVALID, DIR = I, BUS = S_AXI
PORT S_AXI_BREADY = BREADY, DIR = I, BUS = S_AXI
PORT S_AXI_ARADDR = ARADDR, DIR = I, VEC = [(C_S_AXI_ADDR_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_ARVALID = ARVALID, DIR = I, BUS = S_AXI
PORT S_AXI_RREADY = RREADY, DIR = I, BUS = S_AXI
PORT S_AXI_ARREADY = ARREADY, DIR = O, BUS = S_AXI
PORT S_AXI_RDATA = RDATA, DIR = O, VEC = [(C_S_AXI_DATA_WIDTH-1):0], ENDIAN = LITTLE, BUS = S_AXI
PORT S_AXI_RRESP = RRESP, DIR = O, VEC = [1:0], BUS = S_AXI
PORT S_AXI_RVALID = RVALID, DIR = O, BUS = S_AXI
PORT S_AXI_WREADY = WREADY, DIR = O, BUS = S_AXI
PORT S_AXI_BRESP = BRESP, DIR = O, VEC = [1:0], BUS = S_AXI
PORT S_AXI_BVALID = BVALID, DIR = O, BUS = S_AXI
PORT S_AXI_AWREADY = AWREADY, DIR = O, BUS = S_AXI
PORT minimal_leds = "", DIR = O, VEC = [3:0]
END
This file is exactly as generated by the Wizard, except for the HDL option in the beginning changed to VERILOG, and the added port minimal_leds at the end. Note its assignment to “”. This file is best created by looking at examples of existing cores.
Now to minimal_v2_1_0.pao:
lib minimal_v1_00_a minimal verilog
which was rewritten to reflect that the peripheral consists of one single Verilog file.
The device tree file
The device tree file needs to be generated as described in one of my posts. The relevant section is given here, since it relates to kernel code presented next:
minimal_0: minimal@7ae00000 {
compatible = "xlnx,minimal-1.00.a";
reg = < 0x7ae00000 0x10000 >;
xlnx,dphase-timeout = <0x8>;
xlnx,family = "spartan6";
xlnx,num-mem = <0x1>;
xlnx,num-reg = <0x1>;
xlnx,s-axi-min-size = <0x1ff>;
xlnx,slv-awidth = <0x20>;
xlnx,slv-dwidth = <0x20>;
xlnx,use-wstrb = <0x0>;
}
It’s pretty evident that some of these parameters have no use.
The driver
First, it’s convenient to create a makefile for cross compilation. Even though the correct way is to set the environment variables in the shell, and run the module compilation in the same way the kernel itself is compiled, it’s much more convenient to go just “make” or “make clean” with this makefile. It’s not good for distribution, as the paths to both the kernel tree and cross compiler are hardcoded.
So here’s a dirty, but yet convenient makefile:
export CROSS_COMPILE=/path/to/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-
export ARCH=microblaze
ifneq ($(KERNELRELEASE),)
obj-m := minimal.o
else
KDIR := /path/to/linux-2.6.38.6
default:
@echo $(TARGET) > module.target
$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
@rm -f *.ko *.o modules.order Module.symvers *.mod.? .minimal.* *~
@rm -rf .tmp_versions module.target
minimal.ko:
$(MAKE)
endif
And now to the driver itself, minimal.c:
#include <linux/platform_device.h>
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_platform.h>
#include <asm/io.h>
/* Match table for of_platform binding */
static struct of_device_id minimal_of_match[] __devinitdata = {
{ .compatible = "xlnx,minimal-1.00.a", },
{}
};
MODULE_ALIAS("minimal");
static void __iomem *regs;
static struct resource res;
static int __devinit
minimal_of_probe(struct platform_device *op, const struct of_device_id *match)
{
const int *width;
int ret;
int val;
ret = of_address_to_resource(op->dev.of_node, 0, &res);
if (ret) {
printk(KERN_WARNING "minimal: Failed to obtain device tree resource\n");
return ret;
}
printk(KERN_WARNING "minimal: Physical address to resource is %x\n", (unsigned int) res.start);
if (!request_mem_region(res.start, 32, "mimimal")) {
printk(KERN_WARNING "minimal: Failed to request I/O memory\n");
return -EBUSY;
}
regs = of_iomap(op->dev.of_node, 0); /* Verify it's non-null! */
printk(KERN_WARNING "minimal: Access address to registers is %x\n", (unsigned int) regs);
width = of_get_property(op->dev.of_node, "xlnx,slv-dwidth", NULL);
printk(KERN_WARNING "minimal: Obtained width=%d\n", be32_to_cpu(*width));
val = ioread32(regs);
printk(KERN_WARNING "minimal: Read %d, writing %d\n", val, val+1);
iowrite32(++val, regs);
return 0; /* Success */
}
static int __devexit minimal_of_remove(struct platform_device *op)
{
iounmap(regs);
release_mem_region(res.start, 32);
return 0; /* Success */
}
static struct of_platform_driver minimal_of_driver = {
.probe = minimal_of_probe,
.remove = __devexit_p(minimal_of_remove),
.driver = {
.name = "minimal",
.owner = THIS_MODULE,
.of_match_table = minimal_of_match,
},
};
int __init minimal_init(void)
{
int ret;
ret = of_register_platform_driver(&minimal_of_driver);
return ret;
}
void __exit minimal_exit(void)
{
of_unregister_platform_driver(&minimal_of_driver);
}
module_init(minimal_init);
module_exit(minimal_exit);
MODULE_AUTHOR("Eli Billauer");
MODULE_DESCRIPTION("Microblaze minimal module");
MODULE_LICENSE("GPL")
It doesn’t do anything special, except for change the state of the LEDs every time it’s loaded. The drivers also reads one of the parameters from the device tree structure. Not fascinating, but keeps the code, well, minimal.
This code should be pretty straightforward to programmers who are familiar with PCI device drivers, with probing and removal working in more or less the same way. I’ve chosen a hardcoded segment of 32 bytes as the requested region. This depends on the peripheral, of course.
A test run
This is the transcript of the session on the UART console, as run on a freshly booted system. LEDs did indeed go on and off as reflected by the numbers.
/ # insmod minimal.ko
minimal: Physical address to resource is 7ae00000
minimal: Access address to registers is c87e0000
minimal: Obtained width=32
minimal: Read 0, writing 1
/ # lsmod
minimal 1978 0 - Live 0xc8056000
ipv6 305961 10 - Live 0xc8763000
/ # cat /proc/iomem
40600000-4060000f : uartlite
40a00000-40a0ffff : xilinx_spi
40e00000-40e0ffff : xilinx_emaclite
7ae00000-7ae0001f : mimimal
/ # rmmod minimal
rmmod: module 'minimal' not found
/ # cat /proc/iomem
40600000-4060000f : uartlite
40a00000-40a0ffff : xilinx_spi
40e00000-40e0ffff : xilinx_emaclite
/ # lsmod
ipv6 305961 10 - Live 0xc8763000
/ # insmod minimal.ko
minimal: Physical address to resource is 7ae00000
minimal: Access address to registers is c8820000
minimal: Obtained width=32
minimal: Read 1, writing 2
Note that rmmod produces an error message, which makes it look as if it failed to remove the module, when in fact all went well.
The physical address was indeed detected correctly (see device tree), and mapped to another kernel virtual address each time.
This is part IV of my HOWTO on running Linux on Microblaze. The outline is as follows:
Compiling user space applications
We shall now look at how to compile applications for execution under the Microblaze Linux machine. This is pretty straightforward for programs written for the specific environment. The problems may occur when compiling sources which were originally written for fullblown computers, as the build system may not have taken cross compilation into account. And as software projects tend to be hacked to death, with new features added all the time, the code may depend on libraries which are installed on every desktop, but not necessarily on an embedded system. These dependencies are at times a result of a completely offbeat feature, but it’s often simpler to compile the necessary library than to remove the feature. Since there is no single recipe for solving that kind of problems, we’ll stick to the basics of compiling for user space.
Background
Cross compilation of user space applications is actually more difficult than compiling the kernel, mainly because the kernel itself is, after all, a standalone application. There are a few things to take care of:
- Make sure libraries for dynamic linking are in place in the target runtime filesystem, as well as the dynamic linker itself.
- The compilation should be done against the header files corresponding to the libraries present in the target.
- The linked libraries used during compilation should correspond to those in the target.
- The C Runtime stubs (crt1.o, crti.o, crtbegin.o, crtend.o and crtn.o) should fit the Linux user space environment (different files with similar names are used for Microblaze standalones).
This may sound complicated, but most of the job has already been done. So it all boils down to a few simple things to bare in mind, as shown next.
Preparing the target’s file system
In part II it was shown how to download and extract the cross compiler for Microblaze. The same tarball also has the entire package for the target’s root under microblazeel-unknown-linux-gnu/microblazeel-unknown-linux-gnu/sys-root. This directory should be copied into the target’s root as is.
But the target’s root directory is already populated with files as necessary to boot Linux, and run command line utilities with busybox. Some of the files in sys-root, dynamic libraries in particular, already exist in target root, and they’re not identical. But since busybox is statically linked, overwriting the dynamic libraries seems harmless. Overwriting the previous files where applicable is therefore the way to resolve these conflicts, since dynamically linked applications will be compiled against the newer libraries.
All in all, the sys-root directory is ~164 MB, which isn’t too bad when stored on a flash memory. ~129 of these are /lib. The library files include libm, libpthread, libresolv and several other important libraries.
/usr/include takes up ~14 MB, which is probably not necessary on the target system.
Compilation
For cross compilation we use the same compiler used for the kernel. To compile an application:
/path/to/microblazeel-unknown-linux-gnu-gcc --sysroot=/path/to/nfsroot/ -o hello hello.c -lm
where /path/to/nfsroot is the directory which will be the root directory when the executable runs, or a copy of it. The truth is I’m not really sure –sysroot is really necessary, but given the pretty wild search the GNU tools do to find include files and libraries, it looks like a good measure to point directly at where these should be found.
Note that if we omit –sysroot, compiling for Microblaze Linux user space is done simply by using the cross compiler normally, and that works too. This happens because the compiler was configured to look for libraries and includes from its own sys-root. The C runtime stubs are always taken from the compiler’s own.
This is a good time to repeat something said in part II: There is no need to “install” the Microblaze compiler, and neither do the files need to be owned by root. A simple untar anywhere is fine. The compiler uses relative paths to find its resources.
Static linking
To be somewhat safer, static compilation may be preferred, in particular when the whole system’s functionality consists of a single application. The executable file is considerably larger, but doesn’t depend on libraries, so it works even without the sys-root copy mentioned above. Just add the –static flag to gcc. e.g.
/path/to/microblazeel-unknown-linux-gnu-gcc --static --sysroot=/path/to/nfsroot/ -o hello hello.c -lm
The -lm flag is here to demonstrate that libraries should be at the end of the command line. This has no significance when the executable is compiled to be dynamic, but for static linking, failing to put the -lm (and other loadables) at the end will cause misleading errors such as:
/tmp/cckqzZqo.o: In function `main':
: undefined reference to `sin'
collect2: ld returned 1 exit statu
So put the -lm at the end, OK?
Compiling from SDK
Note: I don’t present a working solution in this subject.
Since the BSP, which is generated by the SDK along with the compilation directories, are just include files and libraries, it was appealing to try compiling Linux user space applications from the SDK. The method suggested is to overwrite the BSP created by SDK with include files and libraries for the Linux environment.
The problem I didn’t bother to solve in the end, was that the C runtime libraries used by the linker remained those for a standalone application, so the executable couldn’t run on Linux. Most likely, this can be solved easily, but since I don’t like IDEs myself, I left this issue as is.
Another problem with this method is that the BSP is erased every time the project is rebuilt (but it survives recompilation, of course). So it’s best to keep a copy of the entire BSP directory structure.
In short, this is the procedure, minus the part that makes the linker work with the Linux-related CRTs.
- Create a BSP by creating a C project
- Copy all .a files from sys-root’s /usr/lib and replace the lib directory in the BSP with a directory containing these (only)
- Replace the BSP’s include directory with sys-root’s /usr/include as is
- In SDK (Eclipse, actually), right-click the C project on the Project Explorer and pick Properties. Under C/C++ Build > Settings > Microblaze gcc linker > Linker script, remove the linker script given (edit the text, and remove the part saying ../src/lscript.ld)
And again, this almost works. If someone takes this to the finish line, please let me know.
The setting
I got an AudioCodes MP202B as a phone line adapter from my Israeli ISP, Netvision. The normal way to connect it is putting it between the computer and the ADSL modem, so it does the “dialing” (sending username and password). This was a no-no for me, because I have a little home network with my own NAT and fake DNSes, so the last thing I wanted was to reconfigure my own network.
My twisted, and not really optimal solution was to let the phone adapter think my own computer is the ISP, so it connects to my computer with pppoe, gets a bogus IP address and DNS details, and then connect to the VoIP network through my computer.
That means, among others, that the phone adapter’s packets undergo NAT just like anything else going through that computer. What about incoming calls, I asked myself. They are initiated by the far end. How will iptables know the that its address needs to be mangled, so it goes to the phone adapter, and not the the host?
The answer, as usual with iptables, is don’t worry, be happy. As it turns out, the phone keeps sending initiation packets on UDP port 5060 periodically, so iptables can easily see the session. The voice packets also find their way. In short, it simply works. As usual.
Note that the bogus IP address the adapter sees is exposed to the VoIP operator in the headers of connection establishment packets. So if the ISP suddenly decides to check if the IP address appearing in these packets is in the valid range, the trick is revealed. Actually, it’s enough to check if the IP address in the headers matches the source address of the UDP packet itself (which has been altered by iptables to the real address given by the ISP). A software update or change in the (security?) configuration in the ISP’s infrastructure can lead to a sudden disconnection of the phone line. But this is not likely to happen: Assuming that the ISP checks the phone number against the ISP’s login name, this leaves no room for malicious tricks. Any extra restrictions are unnecessary, and as any network maintainer knows, add any extra filter, and some manager will shout at you soon.
This solution has an inherent flaw, though: Putting the phone between the computer and ADSL modem allows it to prioritize its own packets over data packets. Without this, voice quality can go down as a result of a massive upload to the web. But running a few tests, I didn’t hear any difference.
Getting the link up
Connect the phone adapter through its WAN jack to the desired Ethernet card (eth3 in my case).
Run a sniffer on the port, and verify that the adapter attempts to start a pppoe session. Something like this:
No. Time Source Destination Protocol Info
1 0.000000 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
2 0.999256 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
3 2.999042 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
4 6.998632 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
Set up /etc/ppp/pppoe-server-options, which has the options to pppd, to read something like this:
lcp-echo-interval 10
lcp-echo-failure 2
noauth
ms-dns 10.2.0.1
ms-dns 10.2.0.2
with emphasis on the “noauth” option, since it’s pretty obvious that whoever is connected to the Ethernet jack doesn’t need authentication. Otherwise, the login name and password configured in the phone adapter must be added to pap-secrets or chap-secrets (whichever applies).
The ms-dns option contains DNS addresses for the adapter. These are fake addresses, which are NATed in the hosting machine, so they are real DNSes to the adapter.
Start off the pppoe server (as root, of course) with
# pppoe-server -I eth3 -L 10.192.0.0 -R 10.192.0.1 -N 1
where -N 1 limits the server to only one connections. The -L and -R set the local and remote addresses.
And by the way, to kill the pppoe server along with its connections go:
# killall pppoe-server
The packet capture should now look like:
No. Time Source Destination Protocol Info
1 0.000000 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
2 0.000369 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPPoED Active Discovery Offer (PADO)
3 0.000650 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPPoED Active Discovery Request (PADR)
4 0.001711 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPPoED Active Discovery Session-confirmation (PADS)
5 1.020712 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Request
6 1.021534 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Request
7 1.021687 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Reject
8 1.023117 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Ack
9 1.024261 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Request
10 1.024434 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Ack
11 1.024513 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
12 1.024565 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Request
13 1.025981 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Request
14 1.026087 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Reply
15 1.026740 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Request
16 1.026854 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Nak
17 1.028069 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Request
18 1.028197 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Request
19 1.028267 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Reject
20 1.029948 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
21 1.031115 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Ack
22 1.036625 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Request
23 1.037321 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Ack
24 1.038305 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Ack
25 1.040164 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Request
26 1.043146 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Ack
27 1.572454 10.192.0.1 10.2.0.1 DNS Standard query A ntp.netvision.net.il
28 3.070230 10.192.0.1 10.2.0.1 DNS Standard query A centrex.res.netvision.net.il
29 5.584155 10.192.0.1 10.2.0.1 DNS Standard query A ntp.netvision.net.il
30 6.570551 10.192.0.1 10.2.0.2 DNS Standard query A ntp.netvision.net.il
31 7.019219 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Request
32 7.019401 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Reply
33 8.070333 10.192.0.1 10.2.0.2 DNS Standard query A centrex.res.netvision.net.il
34 8.541665 10.192.0.1 10.2.0.1 DNS Standard query A centrex.res.netvision.net.il
35 10.580118 10.192.0.1 10.2.0.2 DNS Standard query A ntp.netvision.net.il
36 11.030538 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
37 11.031304 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
38 13.018511 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Request
(...etc)
So what we have here is a successful pppoe establishment. It’s also clear that the adapter got the DNS addresses OK, since it uses them for queries. But alas, no answer is returned, because my firewall rejects packets from any ppp device which are not within a session.
On my computer, the firewall script is run every time a ppp device goes up, by virtue of /etc/ppp/ip-up.local calling the firewall setup script.
In the script, I added the following part:
if [ $PHONEIF ] ; then
iptables -A INPUT -i $PHONEIF -j droplog
iptables -A OUTPUT -o $PHONEIF -j droplog
if [ $EXTIF ] ; then
iptables -A FORWARD -i $PHONEIF -o $EXTIF -j ACCEPT
iptables -A FORWARD -i $EXTIF -o $PHONEIF -j ACCEPT
fi
# Default rule: Drop forwarded packets from and to adapter
iptables -A FORWARD -i $PHONEIF -j droplog
iptables -A FORWARD -o $PHONEIF -j droplog
fi
Where $PHONEIF and $EXTIF are the interfaces (ppp1 and ppp0, usually), as defined previously in the script.
Now everything works properly, packet capture as follows:
No. Time Source Destination Protocol Info
1 0.000000 AUDIO_e5:43:0f ff:ff:ff:ff:ff:ff PPPoED Active Discovery Initiation (PADI)
2 0.000059 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPPoED Active Discovery Offer (PADO)
3 0.000319 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPPoED Active Discovery Request (PADR)
4 0.000683 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPPoED Active Discovery Session-confirmation (PADS)
5 0.002173 0.0.0.0 255.255.255.255 DHCP DHCP Discover - Transaction ID 0x1fa2870d
6 1.024784 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Request
7 1.026219 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Request
8 1.026369 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Reject
9 1.027837 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Ack
10 1.028997 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Configuration Request
11 1.029168 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Configuration Ack
12 1.029244 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
13 1.029298 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Request
14 1.031055 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Request
15 1.031172 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Nak
16 1.032347 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Request
17 1.032473 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Request
18 1.032544 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Reject
19 1.034022 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
20 1.035204 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Ack
21 1.040019 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP IPCP PPP IPCP Configuration Request
22 1.041032 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP IPCP PPP IPCP Configuration Ack
23 1.041705 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Ack
24 1.042895 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP CCP PPP CCP Configuration Request
25 1.046591 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP CCP PPP CCP Configuration Ack
26 3.073250 10.192.0.1 10.2.0.1 DNS Standard query A centrex.res.netvision.net.il
27 3.089492 10.2.0.1 10.192.0.1 DNS Standard query response A 82.166.210.6
28 3.106339 10.192.0.1 82.166.210.6 SIP Request: REGISTER sip:centrex.res.netvision.net.il
29 3.131676 82.166.210.6 10.192.0.1 SIP Status: 200 Ok
30 11.034587 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
31 11.035375 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
32 21.034734 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
33 21.035485 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
34 31.034857 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
(etc...)
With SIP registration repeated every ~80 seconds, and LCP echoes every 10 seconds. The SIP protocol is defined in its RFC.
Note that there is no authentication whatsoever. If there was, we would have seen the server sending a challenge, to which the phone adapter would respond with an answer. In the case above, the server accepts the connection with no questions asked.
Registration packets in detail
This is a good time to mention, that I’ve replaced my real incoming phone number to 073-6666666 and calling number to 04-8222222. After all, I don’t want my real phone numbers out there. I’m not customer support.
The Register/OK pair above look like this:
User Datagram Protocol, Src Port: 5060 (5060), Dst Port: 5060 (5060)
Source port: 5060 (5060)
Destination port: 5060 (5060)
Length: 633
Checksum: 0x1623 (correct)
Session Initiation Protocol
Request line: REGISTER sip:centrex.res.netvision.net.il SIP/2.0
Message Header
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e90-100c00a-13c4-50029-0-7e817330-0
To: <sip:200012972736666666@centrex.res.netvision.net.il>
Call-ID: 100b62d8-100c00a-13c4-50029-0-2a161072-0
CSeq: 1 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;rport;branch=z9hG4bK-0-2a0-3639eac2
Max-Forwards: 70
Supported: replaces,100rel
Allow: REGISTER, INVITE, ACK, BYE, REFER, NOTIFY, CANCEL, INFO, OPTIONS, PRACK, SUBSCRIBE
Expires: 1800
Contact: <sip:200012972736666666@10.192.0.1:5060>
User-Agent: MP202 B 2FXS/3.0.1_p041_build_19
Content-Length: 0
and then answer is simply
Session Initiation Protocol
Status line: SIP/2.0 200 Ok
Message Header
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e90-100c00a-13c4-50029-0-7e817330-0
To: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=SDq3rh799-
Call-ID: 100b62d8-100c00a-13c4-50029-0-2a161072-0
CSeq: 1 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;received=46.116.190.192;branch=z9hG4bK-0-2a0-3639eac2;rport=5060
Contact: <sip:200012972736666666@10.192.0.1:5060>;expires=120
Content-Length: 0
Incoming phone call
An incoming phone call, which isn’t answered looks like this:
64 83.972831 82.166.210.6 10.192.0.1 SIP/SDP Request: INVITE sip:200012972736666666@10.192.0.1:5060, with session description
65 84.194196 82.166.210.6 10.192.0.1 SIP/SDP Request: NOTIFY sip:200012972736666666@10.192.0.1:5060, with session description
66 84.339229 10.192.0.1 82.166.210.6 SIP Status: 180 Ringing
67 84.581309 10.192.0.1 82.166.210.6 SIP Status: 200 OK
68 91.035533 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
69 91.036415 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
70 101.035711 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
71 101.036461 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
72 104.484595 82.166.210.6 10.192.0.1 SIP Request: CANCEL sip:200012972736666666@10.192.0.1:5060
73 104.485130 82.166.210.6 10.192.0.1 SIP/SDP Request: NOTIFY sip:200012972736666666@10.192.0.1:5060, with session description
74 104.508747 10.192.0.1 82.166.210.6 SIP Status: 200 OK
75 104.603031 10.192.0.1 82.166.210.6 SIP Status: 487 Request Terminated
76 104.624033 82.166.210.6 10.192.0.1 SIP Request: ACK sip:200012972736666666@10.192.0.1:5060
77 104.629209 10.192.0.1 82.166.210.6 SIP Status: 200 O
The INVITE packet’s details are as follows:
User Datagram Protocol, Src Port: 5060 (5060), Dst Port: 5060 (5060)
Source port: 5060 (5060)
Destination port: 5060 (5060)
Length: 880
Checksum: 0xe69c (correct)
Session Initiation Protocol
Request line: INVITE sip:200012972736666666@10.192.0.1:5060 SIP/2.0
Message Header
Via: SIP/2.0/UDP 82.166.210.6:5060;branch=z9hG4bKckpn1l30a880su8va740.1
Call-Id: SDau8fa01-e00a9a6ae9b1553d6117b50ab6a925a6-a0fo130
From: <sip:048222222@centrex.res.netvision.net.il:5060>;tag=SDau8fa01-10.60.20.110-4294963135-9019
To: <sip:200012972736666666@centrex.res.netvision.net.il:5060>
Max-Forwards: 69
Allow: REGISTER, INVITE, BYE, ACK, CANCEL, REFER, INFO, OPTIONS, SUBSCRIBE, UPDATE
Session-Expires: 900
CSeq: 3040744 INVITE
Contact: <sip:048222222@82.166.210.6:5060;transport=udp>
Supported: timer
Content-Type: application/sdp
Content-Length: 24
Note that the caller’s ID is there!
Outgoing phone call
Picking up the VoIP phone and calling 04-8222222, without the other side answering, yields:
118 221.029993 10.192.0.1 82.166.210.6 SIP/SDP Request: INVITE sip:048222222@centrex.res.netvision.net.il, with session description
119 221.037083 EDIMAX_89:ae:12 AUDIO_e5:43:0f PPP LCP PPP LCP Echo Request
120 221.037885 AUDIO_e5:43:0f EDIMAX_89:ae:12 PPP LCP PPP LCP Echo Reply
121 221.056671 82.166.210.6 10.192.0.1 SIP Status: 100 Trying
122 221.975013 82.166.210.6 10.192.0.1 SIP/SDP Status: 183 Session Progress, with session description
123 222.013388 10.192.0.1 82.166.210.134 UDP Source port: 5004 Destination port: 16480
124 222.032360 10.192.0.1 82.166.210.134 UDP Source port: 5004 Destination port: 16480
125 222.046283 82.166.210.134 10.192.0.1 UDP Source port: 16480 Destination port: 5004
126 222.052372 10.192.0.1 82.166.210.134 UDP Source port: 5004 Destination port: 16480
127 222.066521 82.166.210.134 10.192.0.1 UDP Source port: 16480 Destination port: 5004
128 222.072316 10.192.0.1 82.166.210.134 UDP Source port: 5004 Destination port: 16480
129 222.086438 82.166.210.134 10.192.0.1 UDP Source port: 16480 Destination port: 5004
(and packets keep flowing...)
As people in the industry know, the voice circuit starts without waiting for the other side to answer in an outgoing call.
The invitation packet in detail is pretty much like the previous one, just the other way around:
User Datagram Protocol, Src Port: 5060 (5060), Dst Port: 5060 (5060)
Source port: 5060 (5060)
Destination port: 5060 (5060)
Length: 988
Checksum: 0x196d (correct)
Session Initiation Protocol
Request line: INVITE sip:048222222@centrex.res.netvision.net.il SIP/2.0
Message Header
From: "0736666666"<sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a7b30-100c00a-13c4-50029-da-174234ad-da
To: <sip:048222222@centrex.res.netvision.net.il>
Call-ID: 100b2350-100c00a-13c4-50029-da-4cc1ed17-da
CSeq: 1 INVITE
Via: SIP/2.0/UDP 10.192.0.1:5060;rport;branch=z9hG4bK-da-3561c-7fd18a32
Max-Forwards: 70
Supported: replaces,100rel
User-Agent: MP202 B 2FXS/3.0.1_p041_build_19
Allow: REGISTER, INVITE, ACK, BYE, REFER, NOTIFY, CANCEL, INFO, OPTIONS, PRACK, SUBSCRIBE
Contact: <sip:200012972736666666@10.192.0.1:5060>
Content-Type: application/sdp
Content-Length: 32
Followed by the Trying packet:
User Datagram Protocol, Src Port: 5060 (5060), Dst Port: 5060 (5060)
Source port: 5060 (5060)
Destination port: 5060 (5060)
Length: 371
Checksum: 0x5d52 (correct)
Session Initiation Protocol
Status line: SIP/2.0 100 Trying
Message Header
From: "0736666666"<sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a7b30-100c00a-13c4-50029-da-174234ad-da
To: <sip:048222222@centrex.res.netvision.net.il>
Call-ID: 100b2350-100c00a-13c4-50029-da-4cc1ed17-da
CSeq: 1 INVITE
Via: SIP/2.0/UDP 10.192.0.1:5060;received=46.116.190.192;branch=z9hG4bK-da-3561c-7fd18a32;rport=506
and then
Session Initiation Protocol
Status line: SIP/2.0 183 Session Progress
Message Header
From: "0736666666"<sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a7b30-100c00a-13c4-50029-da-174234ad-da
To: <sip:048222222@centrex.res.netvision.net.il>;tag=SD2kim999-10.60.20.110-4294957762-7934
Call-ID: 100b2350-100c00a-13c4-50029-da-4cc1ed17-da
CSeq: 1 INVITE
Via: SIP/2.0/UDP 10.192.0.1:5060;received=46.116.190.192;branch=z9hG4bK-da-3561c-7fd18a32;rport=5060
Contact: <sip:048222222@82.166.210.6:5060;transport=udp>
Content-Type: application/sdp
Content-Length: 199
Session Description Protocol
Session Description Protocol Version (v): 0
Owner/Creator, Session Id (o): - 1227677235 2 IN IP4 82.166.210.134
Owner Username: -
Session ID: 1227677235
Session Version: 2
Owner Network Type: IN
Owner Address Type: IP4
Owner Address: 82.166.210.134
Session Name (s): -
Connection Information (c): IN IP4 82.166.210.134
Connection Network Type: IN
Connection Address Type: IP4
Connection Address: 82.166.210.134
Time Description, active time (t): 0 0
Session Start Time: 0
Session Start Time: 0
Media Description, name and address (m): audio 16480 RTP/AVP 18 101
Media Type: audio
Media Port: 16480
Media Proto: RTP/AVP
Media Format: 18
Media Format: 101
Media Attribute (a): rtpmap:18 G729/8000
Media Attribute Fieldname: rtpmap
Media Attribute Value: 18 G729/8000
Media Attribute (a): sendrecv
Media Attribute (a): rtpmap:101 telephone-event/8000
Media Attribute Fieldname: rtpmap
Media Attribute Value: 101 telephone-event/8000
Media Attribute (a): fmtp:101 0-15
Media Attribute Fieldname: fmtp
Media Attribute Value: 101 0-1
I have snipped off the session description protocol parts from the previous packets. The “Media Port” entry is obviously how the sides expose their UDP ports.
July 2014 update
A month ago, or so, the phone suddenly stopped connecting, and the attempt to register was refused flat. I called their support, they told me they will be working on it, and then the phone came back to life. This is the updated UDP packet dump of the registration. Note that it fails first, and then the client tries again, with an improved request.
REGISTER sip:centrex.res.netvision.net.il SIP/2.0
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e40-100c00a-13c4-50029-b-5b9cd2a-b
To: <sip:200012972736666666@centrex.res.netvision.net.il>
Call-ID: 100b6288-100c00a-13c4-50029-a-4e521aee-a
CSeq: 1 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;rport;branch=z9hG4bK-b-2cec-1e8eaaf
Max-Forwards: 70
Supported: replaces,100rel
Allow: REGISTER, INVITE, ACK, BYE, REFER, NOTIFY, CANCEL, INFO, OPTIONS, PRACK, SUBSCRIBE
Expires: 1800
Contact: <sip:200012972736666666@10.192.0.1:5060>
User-Agent: MP202 B 2FXS/3.0.1_p041_build_19
Content-Length: 0
SIP/2.0 401 Unauthorized
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e40-100c00a-13c4-50029-b-5b9cd2a-b
To: <sip:200012972736666666@centrex.res.netvision.net.il>
Call-ID: 100b6288-100c00a-13c4-50029-a-4e521aee-a
CSeq: 1 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;received=93.173.36.5;branch=z9hG4bK-b-2cec-1e8eaaf;rport=5060
Content-Length: 0
WWW-Authenticate: Digest realm="NcxSip", nonce="62445575"
REGISTER sip:centrex.res.netvision.net.il SIP/2.0
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e40-100c00a-13c4-50029-b-5b9cd2a-b
To: <sip:200012972736666666@centrex.res.netvision.net.il>
Call-ID: 100b6288-100c00a-13c4-50029-a-4e521aee-a
CSeq: 2 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;rport;branch=z9hG4bK-d-3689-2ec8fad1
Max-Forwards: 70
Supported: replaces,100rel
Allow: REGISTER, INVITE, ACK, BYE, REFER, NOTIFY, CANCEL, INFO, OPTIONS, PRACK, SUBSCRIBE
Expires: 1800
Authorization: Digest username="200012972736666666",realm="NcxSip",nonce="62345345",uri="sip:centrex.res.netvision.net.il",response="26e7f74553dae1131cef72c3c90c5b67",algorithm=MD5
Contact: <sip:200012972736666666@10.192.0.1:5060>
User-Agent: MP202 B 2FXS/3.0.1_p041_build_19
Content-Length: 0
SIP/2.0 200 Ok
From: <sip:200012972736666666@centrex.res.netvision.net.il>;tag=100a4e40-100c00a-13c4-50029-b-5b9cd2a-b
To: <sip:200012972736666666@centrex.res.netvision.net.il>
Call-ID: 100b6288-100c00a-13c4-50029-a-4e521aee-a
CSeq: 2 REGISTER
Via: SIP/2.0/UDP 10.192.0.1:5060;received=93.173.36.5;branch=z9hG4bK-d-3689-2ec8fad1;rport=5060
Contact: <sip:200012972736666666@10.192.0.1:5060>;expires=120
Content-Length: 0
As before, numbers and strings that may be specific have been altered in the dump.
This is part III of my HOWTO on running Linux on Microblaze. The outline is as follows:
Generating the ACE file
The ACE file is what the System ACE chip reads from, and programs the FPGA accordingly. It consists of a sequence of JTAG operations for each necessary task: Configure the FPGA itself, load the software into memory, set the software execution entry point, and kick the software off. All is done with JTAG commands, which the System ACE generates as it scans through its ACE file.
So let’s get down to business.
Create a directory to gather the relevant files, and copy the following into it:
- The Tcl script for generating ACE file: Found at ISE_DS/EDK/data/xmd/genace.tcl (relative to the path where Xilinx ISE is installed)
- The bitstream (system.bit) file created by the EDK (explained in part I). Found in the ‘hw’ subdirectory in the export bundle from EDK to SDK. Or just under ‘implementation’ in the processor’s working directory. It’s the same file.
- The kernel ELF file (simpleImage.xilinx, or the unstripped simpleImage.xilinx.unstrip) created by the kernel build system (explained in part II), found in arch/microblaze/boot/ in the kernel source tree.
Open a command shell (Project > Launch Xilinx Shell if you like), change to this directory and go:
xmd -tcl genace.tcl -hw system.bit -elf simpleImage.xilinx -ace linuxmb.ace -board sp605 -target mdm
which generates a lot of junk files (.svf most notably, which contain JTAG commands in a portable format), and eventually the linuxmb.ace is created (any file name is OK).
In the example above, I assumed that the target is the SP605 board. Looking at the genace.tcl script reveals easily which boards are supported. If it isn’t, it’s not such a big deal. The only reason the board matters is because the System ACE needs to know which device in the JTAG chain to talk with plus some programming parameters. The -board flags to this scrips allows setting the options in a “genace option file” (whatever that means). I would hack the script, though. It looks easier. See here for more information.
Writing to the Compact Flash
First and foremost: If you have a compact flash which boots anything to the FPGA, don’t format it unless you really have to. The System ACE chip (by Xilinx) which reads from the flash directly is a bit picky about the file system format. Preferably use the card which came with the development kit.
And this too: If you just bought a 2 GB flash or so in a general electronics store, odds are that you’ll need to format it.
I explain how to format the flash in another post of mine.
Assuming that the flash is formatted OK, copy the ACE file to the Compact Flash’ root directory. Make sure that
- there is no other *.ace file in the root directory
- there is no xilinx.sys in the root directory
It is perfectly OK to have unrelated directories on the flash, so if there are some files on the flash already, I’d suggest creating a directory with just any name (say, “prevroot”) and move everything in the root directory into that one. And then copy the desired ACE file (linuxmb.ace in the example above) into the root directory.
That’s it. The Linux kernel should now boot, but it will complain (the kernel will panic, actually) that it doesn’t have any root filesystem. So…
Setting up the root filesystem
Once the kernel is up, it needs something to mount as a root filesystem, in which it expects to find its init executable and quite a few other files. Xilinx supplies an image of this bundle which were downloaded along with the cross compilers (see part II), in the same directory.
You may recall that I chose to mount root over the network, using NFS. So to create a useful root directory to work with, just change directory to whatever is going to be root (in my case, the one exposed via NFS) and go
zcat /path/to/microblaze_v1.0_le/initramfs_minimal_le.cpio.gz | cpio -i -d -H newc --no-absolute-filenames
This bundle includes a practical set of executables (well, it’s actually a lot of symbolic links to busybox) including vi, watch, dd, grep, gzip, tar, rpm, nc and even httpd (a web server…!). There’s also a rootfs.cpio.gz in the kernel sources when downloaded from Xilinx’ git (linux-2.6-xlnx.git in part II) which I haven’t tried out. But it’s opened in the same way.
You may, of course, compile your own programs, which is discussed in part IV.
There’s no “shutdown” executable, though. There’s “halt” instead.
A test run
Well, plug in the Compact Flash card, turn the power on, and hope to see a green LED blinking, which turns to steady green after a few seconds. When the LED is steady, expect some output on the UART. A typical log for SP605 is given at the end of this post.
At times, the SP605 board’s green LED went on, but nothing runs until SYS_ACE_RESET is pressed (the middle button out of three close to the Compact Flash jack). Looks like a powerup issue.
Is it fast? Is it fast?
This is maybe not such a fair comparison, and still the facts speak for themselves:
On Microblaze @ 75 MHz clock (37 BogoMIPS):
# dd if=/dev/zero of=/dev/null bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (10.0GB) copied, 1058.304486 seconds, 9.7MB/s
# dd if=/dev/zero of=/dev/null bs=512 count=100k
102400+0 records in
102400+0 records out
52428800 bytes (50.0MB) copied, 9.531130 seconds, 5.2MB/s
The same thing on my own computer @ 1.2 GHz (5600 BogoMIPS):
$ dd if=/dev/zero of=/dev/null bs=1M count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 0.941238 s, 11.4 GB/s
$ dd if=/dev/zero of=/dev/null bs=512 count=100k
102400+0 records in
102400+0 records out
52428800 bytes (52 MB) copied, 0.0443318 s, 1.2 GB/s
According to the BogoMIPSes, Microblaze should have been 150 times slower, not 1000 times slower!
A typical boot log
early_printk_console is enabled at 0x40600000
Ramdisk addr 0x00000003, Compiled-in FDT at 0xc03c2348
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.38.6 (eli@localhost.localdomain) (gcc version 4.1.2) #19 Fri Aug 5 16:40:02 IDT 2011
setup_cpuinfo: initialising
setup_cpuinfo: Using full CPU PVR support
cache: wt_msr_noirq
setup_memory: max_mapnr: 0x8000
setup_memory: min_low_pfn: 0xc0000
setup_memory: max_low_pfn: 0xc8000
On node 0 totalpages: 32768
free_area_init_node: node 0, pgdat c04f515c, node_mem_map c05ca000
Normal zone: 256 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 32512 pages, LIFO batch:7
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32512
Kernel command line: console=ttyUL0 ip=::::::dhcp rootfstype=nfs root=/dev/nfs rw nfsroot=10.11.12.13:/shared/nfsroot,tcp
PID hash table entries: 512 (order: -1, 2048 bytes)
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
allocated 655360 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Memory: 123204k/131072k available
SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:32
xlnx,xps-intc-1.00.a #0 at 0xc8000000, num_irq=8, edge=0x60
xlnx,xps-timer-1.00.a #0 at 0xc8004000, irq=7
Heartbeat GPIO at 0xc8008000
microblaze_timer_set_mode: shutdown
microblaze_timer_set_mode: periodic
Console: colour dummy device 80x25
Calibrating delay loop... 37.17 BogoMIPS (lpj=185856)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
devtmpfs: initialized
NET: Registered protocol family 16
PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
XGpio: /axi@0/gpio@40040000: registered
XGpio: /axi@0/gpio@40020000: registered
XGpio: /axi@0/gpio@40000000: registered
vgaarb: loaded
Switching to clocksource microblaze_clocksource
microblaze_timer_set_mode: oneshot
Switched to NOHz mode on CPU #0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
PCI: CLS 0 bytes, default 32
Skipping unavailable RESET gpio -2 (reset)
GPIO pin is already allocated
audit: initializing netlink socket (disabled)
type=2000 audit(0.429:1): initialized
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
fuse init (API version 7.16)
msgmni has been set to 240
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
40600000.serial: ttyUL0 at MMIO 0x40600000 (irq = 6) is a uartlite
console [ttyUL0] enabled
brd: module loaded
loop: module loaded
of:xsysace 41800000.sysace: Xilinx SystemACE revision 1.0.12
of:xsysace 41800000.sysace: capacity: 3980592 sectors
xsa: xsa1
Xilinx SystemACE device driver, major=254
Generic platform RAM MTD, (c) 2004 Simtec Electronics
xilinx_spi 40a00000.spi: at 0x40A00000 mapped to 0xc8080000, irq=0
of:xilinx_emaclite 40e00000.ethernet: Device Tree Probing
Xilinx Emaclite MDIO: probed
of:xilinx_emaclite 40e00000.ethernet: MAC address is now 00:0a:35:49:b2:00
of:xilinx_emaclite 40e00000.ethernet: Xilinx EmacLite at 0x40E00000 mapped to 0xC80A0000, irq=5
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.19.1-ioctl (2011-01-07) initialised: dm-devel@redhat.com
nf_conntrack version 0.5.0 (1925 buckets, 7700 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
Registering the dns_resolver key type
registered taskstats version 1
Sending DHCP requests .
PHY: c0020918:07 - Link is Up - 100/Full
., OK
IP-Config: Got DHCP answer from 10.11.12.13, my address is 10.11.12.155
IP-Config: Complete:
device=eth0, addr=10.11.12.155, mask=255.255.255.0, gw=10.11.12.13,
host=10.11.12.155, domain=, nis-domain=(none),
bootserver=10.11.12.13, rootserver=10.11.12.13VFS: Mounted root (nfs filesystem) on device 0:13.
devtmpfs: mounted
Freeing unused kernel memory: 147k freed
Starting rcS...
++ Mounting filesystem
++ Starting telnet daemon
rcS Complete
/bin/sh: can't access tty; job control turned off
/ # NET: Registered protocol family 10
eth0: no IPv6 routers present
This is part II of my HOWTO on running Linux on Microblaze. The outline is as follows:
Kernel compilation in general
Compiling a Linux kernel traditionally consists of the following steps (some of which are elaborated further below):
- Obtaining a kernel source tree.
- Configure the kernel. Which all in all means to set up a file named “.config” in the kernel source’s root directory.
- Compile actual kernel, ending up with an executable image.
- Compile the post-boot loadable kernel modules.
- Put everything in its place, set up the bootloader
- Pray and boot
When compiling for Microblaze, the process is somewhat different:
- Cross compilation: The compiled binaries run on a processor different from the one doing the compilation.
- Kernel modules are most likely not used at all. They are a bit pointless when the hardware is known in advance, and also add some complexity in setting up the entire system for boot. Besides, modprobe on a Microblaze can take forever.
- The hardware configuration is custom made, and the kernel needs to be informed about it (through the Device Tree Structure)
Downloading kernel sources
Note that all kernels compile for all target architectures. If you download a kernel from Xilinx’ repository, it may have the parts relevant to Xilinx slightly more updated. The emphasis is on “may”.
The “vanilla” kernel (maintained by Linus Torvalds) can be downloaded from the main kernel archive or one of its mirrors. Several other flavors float around, including Xilinx own git
git clone git://git.xilinx.com/linux-2.6-xlnx.git
or Petalogix’ git (after all, they do a lot of maintenance on the Xilinx devices):
git clone git://developer.petalogix.com/linux-2.6-microblaze.git
The question is always which kernel is best. The answer is that it’s a bit of a gamble. It’s usually almost exactly the same piece of software, with git version having the latest changes. That means the latest bug fixes, new drivers, but also the latest, undocumented and undiscovered bugs. Vanilla kernels tend to be more conservative, but the only rule is that there are no rules. So in short, toss a coin and pick one.
Personally, I compiled the kernel which happened to be on my hard disk for other purposes.
Cross compilers
The good news is that there’s no need to compile the GNU tools. As a matter of fact, this part turned out to be surprisingly painless. The cross compiler and binutils binaries + initramfs images can be downloaded with
$ git clone git://git.xilinx.com/xldk/microblaze_v1.0_le.git
$ git clone git://git.xilinx.com/xldk/microblaze_v1.0.git
Choose one, depending on whether you prefer little endian or big endian for your processor. I picked little endian, but there’s one initramfs in the big endian bundle which isn’t there for the little endian set (which only has the “minimal” image).
One of the files fetched by git is microblazeel-unknown-linux-gnu.tar.gz (gzipped tarball) for the little endian version and mb_gnu_tools_bin.tar.bz (bzipped tarball) for big endian. I’ll leave the latter, because I didn’t use it.
There’s no need to install anything, and no need to be root (actually, doing this as root is pretty unwise). Just untar the tarball of your choice in any directory. Tar generates several subdirectories, but we’re after the cross compilers. Or more precisely, to make the kernel build system use them. This boils down to this:
export CROSS_COMPILE=/home/myhomedir/untarred-to/microblazeel-unknown-linux-gnu/bin/microblazeel-unknown-linux-gnu-
First of all, note the dash at the end of the statement. The whole string is a prefix for all compilation commands made by the kernel build system. It is often recommended to set the path to where the compilers are, and then set CROSS_COMPILE to a shorter prefix. I don’t see the point in polluting the overall path. The build environment has no problem with the statement above.
It has also crossed my mind to use the mb-gcc and friends, which are part of the SDK. But that may require another paid-for license, in case different people do the FPGA and software (which usually is the case).
And to wrap this up: If I’ll ever need to build a cross compiler from scratch, I would start with looking at Buildroot (and another page about it) or following this guide (I haven’t tried either, though).
Kernel configuration
Setting this up correctly is a tedious process, and even the most seasoned kernel hackers may not get it right on the first go. If it’s your first time, prepare to spend quite a few hours on this. The less experienced you are with Linux in general, the more time will you need to spend to make an educated guess about your need for each feature offered.
You can try to use my configuration file, or at least start off with it. It was made for against a 2.6.38 kernel, and booted well as shown in part III. Copy the file as .config on the kernel source’s root, and start with oldconfig.
The commands involved are basically (all “make” commands issues at the kernel source’s top directory):
- Clean up everything, including the .config file if present. This is not necessary if you just uncompressed your kernel. It’s actually rarely necessary at all: “make ARCH=microblaze mrproper”. This will delete .config! (I know I just said it).
- Adopt an existing .config file: “make ARCH=microblaze oldconfig”. This is useful in particular when switching to another kernel version or flavor. Only questions about new features are asked. If you downloaded my configuration file, I would suggest not to turn on options that are offered while running oldconfig, unless they are clearly Xilinx related.
- Configure the kernel: “make ARCH=microblaze xconfig”, “make ARCH=microblaze gconfig” or “make ARCH=microblaze menuconfig” (pick one). These applications present the kernel options in a fairly user-friendly manner, and eventually save the result to .config. I recommend xconfig, because it’s graphic and has a search feature, which turns out very useful.
When targeting an embedded platform, the strategy is to enable whatever is necessary in the kernel itself, and not count on kernel modules. A second issue is to eliminate anything unnecessary from the kernel. This is not just a matter of the kernel image’s size and speed, but enabling components which have nothing to do there can cause the kernel compilation to fail, and even worse, the kernel to crash at boot. Each architecture maintains a set of #include headers, and some kernel components may assume certain things that these architecture-dependent parts haven’t caught up with. So the rule that is whatever hasn’t been tested, won’t necessarily work. Enabling an esoteric keyboard driver on a Microblaze processor may very well fail the boot, simply because nobody cares.
In particular, you most likely want to follow these:
- Under Platform Options, set CONFIG_KERNEL_BASE_ADDR to where your DDR RAM begins (0xC0000000 on my processor), the targeted FPGA family as well as the other parameters (USE_*). The USE_* parameters’ correct values can be found in the .dts file. Just copy the values of the processor elements with the same names.
- Also set
CONFIG_SERIAL_UARTLITE=y
CONFIG_SERIAL_UARTLITE_CONSOLE=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
- Since we’re not going to use any boot loader, the kernel command line needs to be compiled into the kernel itself: Enable CMDLINE_BOOL (default bootloader kernel argument) and set it to something useful. As for the console, set it to console=ttyUL0, or nothing goes to console after the two first lines sent to console from early_printk_console (CONFIG_CMDLINE_FORCE may be necessary as well. It doesn’t hurt in the absence of a boot loader anyhow)
- Enable CONFIG_MSDOS_FS and CONFIG_VFAT_FS in kernel (not module), so that the SystemACE can be read.
- Enable CONFIG_XILINX_SYSACE
- Enable CONFIG_XILINX_EMACLITE and CONFIG_FB_XILINX
- Disable the FTRACE config option (under kernel hacking, compilation fails) instead of using patch.
And for your own sake, make a copy of the .config file every now and then as you work on it. It’s very easy to delete it by mistake or to mess it up in general.
Setting the Linux boot parameters correctly is very important, because if they’re wrong, kernel recompilation is they only way to fix it in the absence of a boot loader. I’ve chosen to mount the root directory from the network, but note that /dev/sxa is the Compact flash itself (with /dev/sxa1 is the first partition, for example). So it’s fairly simple to add a partition to the flash device, and put a regular root filesystem there. Maybe I’ll do that myself and update this post.
Anyhow, my choice for the Linux boot parameters was
console=ttyUL0 ip=::::::dhcp rootfstype=nfs root=/dev/nfs rw nfsroot=10.11.12.13:/shared/nfsroot,tcp
where “/shared/nfsroot” is the shared NFS directory on the server with IP 10.11.12.13. This command is suitable for getting the root from the network, which is very convenient for development. This setting requires a DHCP server on the LAN. In case you don’t want to configure a DHCP server, use the ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:::off format instead. Documentation/filesystems/nfs/nfsroot.txt in the kernel sources has more about booting from NFS. I’ve also written a post about booting a diskless PC from network, but it’s a bit of an overkill.
In case you’re interested in how the whole configuration thing comes together, let’s take CONFIG_EARLY_PRINTK for example. In arch/microblaze/kernel/Makefile, one of the lines says:
obj-$(CONFIG_EARLY_PRINTK) += early_printk.o
On the other hand, in the config file it can say
CONFIG_EARLY_PRINTK=y
So when the Makefile is executed, the target early_prink.o is added to either obj-y, obj-m or obj-n. obj-y is the list of objects to be inserted into the kernel, obj-m is the list of modules, and obj- is the junk list. The configuration rules are given in the Kbuild files, next to the Makefiles.
A small Makefile fix
As of 2.6.38, there is a small error in the arch/microblaze/boot/Makefile, which makes the build system always attempt making an U-Boot image, which is not necessary in our case. This may result in an error message (saying “mkimage” wasn’t found), when everything is actually OK. So in the part saying
$(obj)/simpleImage.%: vmlinux FORCE
$(call if_changed,cp,.unstrip)
$(call if_changed,objcopy)
$(call if_changed,uimage)
$(call if_changed,strip)
@echo 'Kernel: $@ is ready' ' (#'`cat .version`')'
remove or comment out the line saying “$(call if_changed,uimage)”.
Compiling the kernel
Before starting: You didn’t forget to set CROSS_COMPILE and copy the updated xilinx.dts file to its place… right?
I prefer cleaning up before compiling:
make ARCH=microblaze clean
rm arch/microblaze/boot/simpleImage.*
This is a good time to ask why the image file isn’t cleaned by “make clean”. To be fixed, I suppose.
And then, the compilation is just
make -j 8 ARCH=microblaze simpleImage.xilinx
Note that the “.xilinx” suffix corresponds to the xilinx.dts file in the arch/microblaze/boot/dts/ directory. If another .dts file should be made effective, change the suffix.
The “-j 8″ means that 8 compilation processes run in parallel, which is suitable for a quad processor with hyperthreading. Skip this option or use another number, depending on your computer, your spare time and your need to see the logic of the events.
The basic UNIX rule is that everything went fine unless an error message appeared. A more explicit confirmation is that it said
OBJCOPY arch/microblaze/boot/simpleImage.xilinx
somewhere close to the end, and that the arch/microblaze/boot/simpleImage.xilinx is indeed there, and has a date stamp that makes sense.
If and when you get errors, well, there’s no simple recipe to solve that. The easiest way is to eliminate the need to compile that certain file by changing the kernel configuration, if the functionality is indeed unnecessary. Otherwise your best friends are Google and your brain, not necessarily in that order.
As for the Device Tree, it was compiled into a .dtb file (the Device Tree binary blob), which can be found in the same directory as the just generated kernel image. The Device Tree Compiler (dtc) comes with the kernel sources, and can be found in scripts/dtc.
And just to wrap this up: If you insist on seeing all the commands issued instead of the otherwise laconic output, there the KBUILD_VERBOSE flag. For example,
make ARCH=microblaze KBUILD_VERBOSE=1 clean
With a compiled kernel image at hand (which already has the Device Tree built-in), all that’s left is to set up the Compact Flash and boot. Go to part III of this HOWTO.
A few other make statements
For completeness:
- Clean up any compiled binaries: Recommended after a change in .config: “make ARCH=microblaze clean”
- Generate loadable modules: “make ARCH=microblaze modules”. Not necessary if everything needed is compiled into the kernel.
- And then gather the modules in a neat directory (making sure you don’t have a /lib/modules directory with the same version number): “make ARCH=microblaze modules_install”. This will write to /lib/modules on the local machine, so if you happen to compile exactly the same kernel version for your own PC and the embedded target, the kernel modules the PC relies on will be overwritten.
This is part I of my HOWTO on running Linux on Microblaze. The outline is as follows:
Introduction
This HOWTO goes through the procedures for getting a simple Linux system running on a Xilinx Microblaze processor. The examples are given for an SP605 evaluation board, but almost everything here applies for other FPGAs and boards as well. The Xilinx software version used here is 13.2.
There are quite a few variants on how to get the bitstream and Linux kernel into their right places in the FPGA. The approach taken here is to boot up from the Compact Flash alone by writing a file to it. No bootloader is used in this howto; the SystemACE chip is responsible for loading both the FPGA bitstream and Linux kernel image, and it will do so reading one single (.ace) file. The main advantage of this approach is that there’s no need to set up a boot loader, which is yet another piece of software that can go wrong. The main disadvantage is that a bootloader allows some tweaking of the kernel configuration at boot time, which has to be done by recompiling the kernel otherwise.
The root filesystem is mounted from network (NFS) in this HOWTO.
I’m assuming the following prerequisites:
- You have the Xilinx tools set up properly, and have managed to compile and run a simple standalone “Hello, World” application with the EDK/SDK (having loaded the code to the FPGA in any way, we’ll get to that)
- You’ve actually seen the RS-232 console data on a terminal, and feel confident about it (otherwise you may work hard to figure out why everything is stuck, when it’s actually your terminal window’s problem).
- You’re running on one of the evaluation boards, or know how to set up the processor to work with your own (and have that tested already)
- Your board has a systemACE flash chip (recent evaluation boards usually do)
- You have access to a machine running Linux on a computer. Compiling the kernel will require this. The Xilinx tools can be run on whatever’s your cup of tea.
- You have the ability to read and write files to a Compact Flash. This is most easily done with a simple adapter to a PC computer, which should be available in camera or computer accessories shops. Chances are you have one without necessarily being aware of it.
An outline of the steps
So this is what we’ll do:
- Set up a Microblaze processor in the Xilinx EDK so it can run Linux.
- Generate the processor, so an FPGA bitstream is at hand.
- Export the processor to the Xilinx SDK and compile a dummy C application, so that necessary metadata files are generated
- Generate a Device Tree file (.dts) based upon files created by EDK/SDK, and copy it into the Linux kernel sources, so Linux is in sync with the EDK regarding what it’s running on.
- Configure the kernel and compile it.
- Create a .ace file from the FPGA bitstream and kernel image just compiled.
- Set up the Compact Flash card.
- Boot and hope for good
And of course, certain software tools will need to be downloaded for this. We’ll come to this.
Setting up the processor
If you’re really lazy about this, you can use the minimal processor I’ve generated for the SP605 board. Unzip, double-click system.xmp, and skip to after the bullets below. It will work on that board only, of course.
Otherwise: Start Platform Studio (EDK) and create a new platform, based upon the Wizard’s defaults.
Following a Microblaze on Linux guide, in particular the part regarding minimal hardware requirements, there a need to make sure that the hardware has an MMU with two regions, a timer, an interrupt controller and a UART with an interrupt line. In the platform studio it goes like this:
Starting off with the Wizard’s defaults,
- Double click “microblaze_0″ on the Ports tab, and set the Linux with MMU preset on the Configuration wizard showing up. This will take care of most settings.
- Still in the ports view, add an AXI Interrupt Controller (under Clock, Reset and Interrupt in the IP Catalog). Accept default settings. Make a new connection for its irq output, and connect it to the microblaze_0′s interrupt input pin.
- Pick the RS232_Uart_1 and make a new connection for the interrupt line. Connect that signal to the interrupt controller.
- Add an AXI Timer/Counter, and accept defaults. Make a new connection for the interrupt, and connect it to the interrupt controller.
- Connect the interrupts of the Ethernet lite, SPI Flash, IIC SFP, IIC EEPROM, IIC_DVI, and SysACE cores to the interrupt controller as well.
Then generate bitstream, export to SDK, and run the SDK, adopting this hardware platform. The goal of this is to generate a .mss file, which will be needed later. For this to happen, make a new C project (“Hello World” will do just fine) and compile it.
There is no need to “update the bitstream” like with standalone applications: The Linux kernel can take care of itself, without having its entry address hardwired in the FPGA’s block RAM. We’ll use the system.bit, and not the download.bit (even though the latter works too).
Creating a Device Tree file
The purpose of this stage is to generate a .dts file, which is the format expected by the kernel build environment. It informs the kernel about the structure of the processor and its peripherals. The device tree structure is discusses further here.
If you chose to download and use my processor with no changes whatsoever, you can also get my DTS file. Just copy it to arch/microblaze/boot/dts/ in the to-be compiled kernel source tree.
To make your own .dts file, first create a special directory, and make it the working directory of your shell.
The device tree file is generated automatically with the libgen utility with the help of a Tcl script. As of ISE 13.2, this script needs to be loaded separately with git:
bash> git clone git://git.xilinx.com/device-tree.git
This generates a device-tree directory. Another web page explains how to make SDK recognize the script, but I prefer command line for things like this. Another post of mine explains the device tree further.
Copy the system.xml file from the directory to which you exported to SDK (in the “hw” subdirectory), into the current one. Then copy system.mss from the project’s BSP directory. It will have a name like hello_world_bsp_0.
Edit the copy you made of system.mss, so that the BEGIN OS to END part reads
BEGIN OS
PARAMETER OS_NAME = device-tree
PARAMETER OS_VER = 0.00.x
PARAMETER PROC_INSTANCE = microblaze_0
END
and not “standalone” for OS.
And then run libgen as follows (make sure it’s in the PATH. The easiest way is to launch a “Xilinx shell” from the EDK’s project menu):
libgen -hw system.xml -lp device-tree -pe microblaze_0 -log libgen.log system.mss
Which generates a xilinx.dts in microblaze_0/libsrc/device-tree_v0_00_x. Copy this file to arch/microblaze/boot/dts/ in the to-be compiled kernel source tree. If you can’t find the file there, and libgen didn’t complain about some error, you may have forgotten to edit system.mss as mentioned just above.
Now let’s go on to compiling the kernel, in part II.
Spoiler
It’s very likely that you don’t need to read this. If all you want is to get a Linux kernel to detect a Microblaze processor on an Xilinx FPGA, the relevant information is in another post of mine. This post goes into the details which are necessary to understand, if you want to write a kernel driver for a device tree mapped peripheral.
Why a device tree is necessary
The main issue with running Linux on an FPGA is that the Linux kernel needs to know what peripherals it has and where it can find them. On PC computers this problem was solved many years ago with the PCI bus: The BIOS detects the peripherals, allocates their addresses and interrupts and tells the operating system what it has and where it can be found. In the embedded world, this information was hardcoded into pieces of the kernel sources, which were written specifically for every board. With many boards out there, the kernel source grew way too fast. This far-from-optimal solution is not feasible with a soft processor, whose peripherals are configured per case. Hacking the kernel sources to match the FPGA is a recipe for bugs, crashes and being stuck with a certain kernel forever.
The elegant solution for this is the Flattened Device Tree. The idea is to create some binary data structure, which is either linked into the kernel image or given to it during boot. This binary blob contains the information about the processor itself and its peripherals, including the addresses, interrupts and several application-specific parameters. So the drivers for these peripherals are written very similar to PCI drivers: They declare what peripherals they support, and obtain their resources from a standard kernel API.
The code for Flattened Device Tree and Open Firmware resides in drivers/of in the kernel tree. The relevant include file is include/linux/of.h.
Generation
Note that at least for Xilinx FPGAs, there is no need to generate the device tree manually. Rather, get a copy of the device tree generator with
bash> git clone git://git.xilinx.com/device-tree.git
which basically consists of a TCL script run by libgen and a configuration file. The device tree generator’s page explains how to make SDK recognize the script, but there’s no reason to play around with SDK for that.
Instead, go
libgen -hw /path/to/system.xml -lp /path/to/device-tree -pe microblaze_0 -log libgen.log system.mss
Which generates a system.dts in microblaze_0/libsrc/device-tree_v0_00_x
The system.mss file is generated as a byproduct when compiling just any a project within SDK, and is found under the directory with the _bsp_n suffix. I still need to find out how to create the file from the command line.
It needs to be modified, so that the BEGIN OS to END part reads
BEGIN OS
PARAMETER OS_NAME = device-tree
PARAMETER OS_VER = 0.00.x
PARAMETER PROC_INSTANCE = microblaze_0
END
and not “standalone” for OS.
To get the system.xml file (which was necessary to create the system.mss), go Project > Export Hardware to SDK in the EDK platform studio. Or
make -f system.make exporttosdk
from the project’s home directory.
The correct setup of the device tree entry can be found in the Documentation/devicetree/bindings directory of the kernel sources. The xilinx.txt file describes the bindings for Xilinx peripherals, and explains how information in the system.mhs file is translated into a xilinx.dts.
As part of a full kernel compilation, the .dts is compiled into a .dtb file (the Device Tree binary blob), which can be found in the same directory as the generated kernel image. The Device Tree Compiler (dtc) comes with the kernel sources, and can be found in scripts/dtc.
A sample entry
The following example is given there for a Uartlite (which we’ll follow on below):
opb_uartlite_0: serial@ec100000 {
device_type = "serial";
compatible = "xlnx,opb-uartlite-1.00.b";
reg = <ec100000 10000>;
interrupt-parent = <&opb_intc_0>;
interrupts = <1 0>; // got this from the opb_intc parameters
current-speed = <d#115200>; // standard serial device prop
clock-frequency = <d#50000000>; // standard serial device prop
xlnx,data-bits = <8>;
xlnx,odd-parity = <0>;
xlnx,use-parity = <0>;
};
It’s recommended to have a look at arch/microblaze/platform/generic/system.dts in the kernel sources for a fullblown file. Or one you’ve generated yourself, for that matter.
Declarations in a kernel module driver
Device tree mapped instances are treated by the kernel very much like PCI devices, only the source of information is the DTB (Device Tree Binary) rather than from the BIOS.
The parallel to PCI’s Vendor/Product IDs is an entry looking like this (taken from uartlite.c):
static struct of_device_id ulite_of_match[] __devinitdata = {
{ .compatible = "xlnx,opb-uartlite-1.00.b", },
{ .compatible = "xlnx,xps-uartlite-1.00.a", },
{}
};
MODULE_DEVICE_TABLE(of, ulite_of_match)
Which is then bound to a driver with
static struct of_platform_driver ulite_of_driver = {
.probe = ulite_of_probe,
.remove = __devexit_p(ulite_of_remove),
.driver = {
.name = "uartlite",
.owner = THIS_MODULE,
.of_match_table = ulite_of_match,
},
}
and then, finally, exposed to the kernel with
static inline int __init ulite_of_register(void)
{
pr_debug("uartlite: calling of_register_platform_driver()\n");
return of_register_platform_driver(&ulite_of_driver);
}
somewhere at the end of the driver’s code. This format is very similar to the declaration of PCI devices, so if this is unclear, I’d suggest learning how to do it the PCI way, which is by far more documented.
And by the way, when the kernel is configured to support it, the device tree can be viewed in human-readable format in /proc/device-tree.
The of_device_id structure
The structure is defined in include/linux/mod_devicetable.h as
struct of_device_id
{
char name[32];
char type[32];
char compatible[128];
#ifdef __KERNEL__
void *data;
#else
kernel_ulong_t data;
#endif
};
Surprisingly enough, the lengths of the entries are fixed and limited.The three strings, name, type and compatible are compared as strings (with strcmp(), see of/base.c) with the device tree’s node’s data. Everything declared (that is, non-NULL) in the structure must be equal with the node’s info for a match. In other words, NULLs are wildcards.
In the declaration example above, only the “compatible” part was declared, so any device matching the string exactly triggers off a probe on the driver.
Introduction
The Xilinx Platform Studio (EDK) has this “update bitstream” function, which I wasn’t so clear about, despite its documentation page. Its icon says “BRAM INIT” which turns out to be more accurate than expected. So what happens during this process? When is it necessary?
If you’re into running a Linux kernel, you’re most likely wasting your time reading this, because the Linux kernel is kicked off directly from the external RAM, and hence this mangling isn’t necessary. To set up a Linux bitstream, see another post of mine.
Having that said, let’s look at the problem this functions solves: A Microblaze processor starts executing at address 0 unless told otherwise. Its interrupt vectors are at near-zero addresses as well. These addresses are mapped to an FPGA block RAM.
What this block RAM should contain is a jump to the application’s entry point. On a SP605 board, this is most likely the beginning of the DDR memory, Oxc0000000. So when the processor kicks off, this block RAM’s address zero should contain:
00000000 <_start>:
0: b000c000 imm -16384
4: b8080000 brai 0
Which is Microblazish for “Jump to Oxc0000000″ (note the lower 16 bits of both commands).
When a system is booted, there are two phases: First, the FPGA is loaded with its bitstream, and then the external memory, containing the bulk of execution code. And then the processor is unleashed.
So the block memory’s correct content needs to be included in the bitstream itself. But when the processor is implemented from its logic elements, it isn’t clear what should be written there. It’s only when the software is linked, that the addresses of the different segments are known.
But software compilation and linking requires the knowledge of the processor’s memory map, which is generated while the processor is implemented. So there’s a chicken-and-egg situation here.
The egg was first
The solution is that block RAM’s content is fixed after the software is compiled and linked. The reset and interrupt vectors are included in the ELF file generated by the software linker, and are mapped to the block RAM’s addresses. The “update bitstream” process reads the ELF file, finds the relevant region, and updates the bitstream file, producing the download.bit file. That’s why choosing the ELF file is necessary for this process.
Necessity
The original problem was that the execution starts from address zero. But if the ELF file points at the real starting point, and this is properly communicated to the processor at startup, there’s no need to set up the block RAM at all. Well, assuming that the executable takes care of interrupts and exception vectors soon enough. This is the case with Linux kernel images, for example, for which there is no need to update the bitstream.
Some gory details
The “update bitstream” process launches a command like
bitinit -p xc6slx45tfgg484-3 system.mhs -pe microblaze_0 sdk/peripheral_tests_0/Debug/peripheral_tests_0.elf \
-bt implementation/system.bit -o implementation/download.bit
which takes place in two phases. In the first phase, the system.mhs file is read and parsed, so that the memory map is known and the block RAM is identified. This program then runs something like
data2mem -bm "implementation/system_bd" -p xc6slx45tfgg484-3 -bt "implementation/system.bit" -bd "sdk/peripheral_tests_0/Debug/peripheral_tests_0.elf" tag microblaze_0 -o b implementation/download.bit
Which is the action itself. Data2mem is a utility for mangling bitstreams so that their block RAMs contain desired data. The -bm flag tells data2mem to get the block RAM map from implementation/system_bd.bmm, which can be
// BMM LOC annotation file.
//
// Release 13.2 - Data2MEM O.61xd, build 2.2 May 20, 2011
// Copyright (c) 1995-2011 Xilinx, Inc. All rights reserved.
///////////////////////////////////////////////////////////////////////////////
//
// Processor 'microblaze_0', ID 100, memory map.
//
///////////////////////////////////////////////////////////////////////////////
ADDRESS_MAP microblaze_0 MICROBLAZE-LE 100
///////////////////////////////////////////////////////////////////////////////
//
// Processor 'microblaze_0' address space 'microblaze_0_bram_block_combined' 0x00000000:0x00001FFF (8 KBytes).
//
///////////////////////////////////////////////////////////////////////////////
ADDRESS_SPACE microblaze_0_bram_block_combined RAMB16 [0x00000000:0x00001FFF]
BUS_BLOCK
microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_0 [31:24] INPUT = microblaze_0_bram_block_combined_0.mem PLACED = X3Y30;
microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_1 [23:16] INPUT = microblaze_0_bram_block_combined_1.mem PLACED = X2Y30;
microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_2 [15:8] INPUT = microblaze_0_bram_block_combined_2.mem PLACED = X2Y32;
microblaze_0_bram_block/microblaze_0_bram_block/ramb16bwer_3 [7:0] INPUT = microblaze_0_bram_block_combined_3.mem PLACED = X2Y36;
END_BUS_BLOCK;
END_ADDRESS_SPACE;
END_ADDRESS_MAP;
So this file defines the addresses covered as well as the physical positions of these block RAMs in the logic fabric.
The -bd flag points at the ELF file to get the data from, with the “tag microblaze_0″ part saying that only the memories tagged microblaze_0 in the .bmm file should be handled, and the rest ignored.
This is a small reverse-engineering of the ELF file, as generated by Xilinx’ SDK for a simple standalone application targeted for the SP605 board.
ELF headers
Looking into the ELF file, we have something like this:
> mb-objdump --headers sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf
sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf: file format elf32-microblazele
Sections:
Idx Name Size VMA LMA File off Algn
0 .vectors.reset 00000008 00000000 00000000 000000b4 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .vectors.sw_exception 00000008 00000008 00000008 000000bc 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .vectors.interrupt 00000008 00000010 00000010 000000c4 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
3 .vectors.hw_exception 00000008 00000020 00000020 000000cc 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
4 .text 0000653c c0000000 c0000000 000000d4 2**2
CONTENTS, ALLOC, LOAD, CODE
5 .init 0000003c c000653c c000653c 00006610 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
6 .fini 0000001c c0006578 c0006578 0000664c 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
7 .ctors 00000008 c0006594 c0006594 00006668 2**2
CONTENTS, ALLOC, LOAD, DATA
8 .dtors 00000008 c000659c c000659c 00006670 2**2
CONTENTS, ALLOC, LOAD, DATA
9 .rodata 00000986 c00065a4 c00065a4 00006678 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
10 .sdata2 00000006 c0006f2a c0006f2a 00006ffe 2**0
ALLOC
11 .sbss2 00000000 c0006f30 c0006f30 000071d8 2**0
CONTENTS
12 .data 000001d0 c0006f30 c0006f30 00007000 2**2
CONTENTS, ALLOC, LOAD, DATA
13 .eh_frame 00000004 c0007100 c0007100 000071d0 2**2
CONTENTS, ALLOC, LOAD, DATA
14 .jcr 00000004 c0007104 c0007104 000071d4 2**2
CONTENTS, ALLOC, LOAD, DATA
15 .sdata 00000000 c0007108 c0007108 000071d8 2**0
CONTENTS
16 .sbss 00000000 c0007108 c0007108 000071d8 2**0
CONTENTS
17 .tdata 00000000 c0007108 c0007108 000071d8 2**0
CONTENTS
18 .tbss 00000000 c0007108 c0007108 000071d8 2**0
19 .bss 00000d78 c0007108 c0007108 000071d8 2**2
ALLOC
20 .heap 00000400 c0007e80 c0007e80 000071d8 2**0
ALLOC
21 .stack 00000400 c0008280 c0008280 000071d8 2**0
ALLOC
22 .debug_line 0000779f 00000000 00000000 000071d8 2**0
CONTENTS, READONLY, DEBUGGING
23 .debug_info 00008b11 00000000 00000000 0000e977 2**0
CONTENTS, READONLY, DEBUGGING
24 .debug_abbrev 000028e7 00000000 00000000 00017488 2**0
CONTENTS, READONLY, DEBUGGING
25 .debug_aranges 000006c0 00000000 00000000 00019d70 2**3
CONTENTS, READONLY, DEBUGGING
26 .debug_macinfo 0007f541 00000000 00000000 0001a430 2**0
CONTENTS, READONLY, DEBUGGING
27 .debug_frame 00000f10 00000000 00000000 00099974 2**2
CONTENTS, READONLY, DEBUGGING
28 .debug_loc 00003f80 00000000 00000000 0009a884 2**0
CONTENTS, READONLY, DEBUGGING
29 .debug_pubnames 00000fbe 00000000 00000000 0009e804 2**0
CONTENTS, READONLY, DEBUGGING
30 .debug_str 000018d5 00000000 00000000 0009f7c2 2**0
CONTENTS, READONLY, DEBUGGING
31 .debug_ranges 00000078 00000000 00000000 000a1097 2**0
CONTENTS, READONLY, DEBUGGING
Even though this is a lot of mumbo-jumbo, there are three main parts. The reset and interrupt vectors, around address zero, the main parts of the ELF (.text, .data and such) at Oxc0000000 and on, and the debug parts which have no memory allocation at all.
The reset branch to application
This is interesting to compare with the Microblaze’s memory map. It can be deduced from the .mhs file, but hey, the log file (with .log suffix) has this segment:
Address Map for Processor microblaze_0
(0000000000-0x00001fff) microblaze_0_d_bram_ctrl microblaze_0_dlmb
(0000000000-0x00001fff) microblaze_0_i_bram_ctrl microblaze_0_ilmb
(0x40000000-0x4000ffff) Push_Buttons_4Bits axi4lite_0
(0x40020000-0x4002ffff) LEDs_4Bits axi4lite_0
(0x40040000-0x4004ffff) DIP_Switches_4Bits axi4lite_0
(0x40600000-0x4060ffff) RS232_Uart_1 axi4lite_0
(0x40800000-0x4080ffff) IIC_SFP axi4lite_0
(0x40820000-0x4082ffff) IIC_EEPROM axi4lite_0
(0x40840000-0x4084ffff) IIC_DVI axi4lite_0
(0x40a00000-0x40a0ffff) SPI_FLASH axi4lite_0
(0x40e00000-0x40e0ffff) Ethernet_Lite axi4lite_0
(0x41800000-0x4180ffff) SysACE_CompactFlash axi4lite_0
(0x74800000-0x7480ffff) debug_module axi4lite_0
(0xc0000000-0xc7ffffff) MCB_DDR3 axi4_0
So obviously all the main ELF parts go directly to the DDR memory (that isn’t much of a surprise), and the reset/interrupt go to the internal block ram.
A quick disassembly reveals the gory details:
> mb-objdump --disassemble sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf
sdk/peripheral_tests_1/Debug/peripheral_tests_1.elf: file format elf32-microblazele
Disassembly of section .vectors.reset:
00000000 <_start>:
0: b000c000 imm -16384
4: b8080000 brai 0
Disassembly of section .vectors.sw_exception:
00000008 <_vector_sw_exception>:
8: b000c000 imm -16384
c: b8081858 brai 6232
Disassembly of section .vectors.interrupt:
00000010 <_vector_interrupt>:
10: b000c000 imm -16384
14: b80818a4 brai 6308
Disassembly of section .vectors.hw_exception:
00000020 <_vector_hw_exception>:
20: b000c000 imm -16384
24: b8081870 brai 6256
Disassembly of section .text:
c0000000 <_start1>:
c0000000: b000c000 imm -16384
c0000004: 31a07108 addik r13, r0, 28936
c0000008: b000c000 imm -16384
c000000c: 30406f30 addik r2, r0, 28464
(... and it goes on and on ...)
So let’s look at the reset vector at address zero. The first IMM opcode loads C000 as the upper 16 bits for the command following, which is a branch immediate command. Together, they make a jump to Oxc000000. Likewise, the software exception jumps to Oxc0001858 and so on.
Since only the block RAM part can be included in the download.bit bitfile, only these jump vectors depend on the ELF file during the “Update bitfile” process. That’s why one gets away with not running this process, even when the ELF has been modified with a plain recompilation.
And now to the bootloop ELF
So what is the bootloop code doing? The headers are no more impressive than
> mb-objdump --headers bootloops/microblaze_0.elf
bootloops/microblaze_0.elf: file format elf32-microblazele
Sections:
Idx Name Size VMA LMA File off Algn
0 .boot 00000004 00000000 00000000 00000074 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .text 00000000 00000000 00000000 00000074 2**0
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .data 00000000 00000000 00000000 00000074 2**0
CONTENTS, ALLOC, LOAD, DATA
3 .bss 00000000 00000000 00000000 00000078 2**0
ALLOC
Note the Size column: All entries are empty, except for the .boot section, which is four bytes small (one single instruction). That doesn’t leave room for sophisticated software, and the disassembly is indeed
> mb-objdump --disassemble bootloops/microblaze_0.elf
bootloops/microblaze_0.elf: file format elf32-microblazele
Disassembly of section .boot:
00000000 <_boot>:
0: b8000000 bri 0 // 0
Which is simply an endless loop. So they called it bootloop for a reason.
This is a small guide to loading a standalone application + bitstream to an FPGA using the CompactFlash card. Or put otherwise, how to make the System ACE chip happy.
For loading a Linux kernel in the same way, I suggest referring to a special post in that subject.
Formatting the flash
Rule #1: Don’t format it unless you have to. And if you have to, read the System ACE CompactFlash Solution datasheet (DS080.pdf), in particular “System ACE CF Formatting Requirements” which basically says that if you format the flash under XP, it won’t work. To summarize it shortly,
- Make it a FAT12 or FAT16, and not a FAT32 (the usual choice)
- More than one sector per cluster
- Only one reserved sector (XP may very well allocate more)
- Maximum 2GB capacity (note that when it says 2GB commercially, it’s usually slightly less, but can be more. Partitioning is recommended)
It’s recommended to rewrite the partition table, as it may arrive messy. With fdisk, this is a desired final format (give or take sizes):
Disk /dev/sdd: 2017 MB, 2017419264 bytes
64 heads, 63 sectors/track, 977 cylinders
Units = cylinders of 4032 * 512 = 2064384 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdd1 1 977 1969600+ 6 FAT16
NOTE: My Flash Disk appeared as /dev/sdd, yours may appear as something else. Don’t forget to fix this when running these commands, or you may wipe your hard disk!
Note the file system ID 6 (FAT16). The card originally arrived with type 4, which is “FAT16 < 32MB”. To format the Compact Flash correctly in Linux, go (change sdd1 with the correct device, or erase something you didn’t want to):
# mkdosfs -R 1 -F 16 /dev/sdd1
And then verify that you got one single reserved sector (it’s likely you got it wrong):
# hexdump -n 32 -C /dev/sdd1
00000000 eb 3c 90 6d 6b 64 6f 73 66 73 00 00 02 20 01 00 |.<.mkdosfs... ..|
00000010 02 00 02 00 00 f8 f5 00 3f 00 40 00 00 00 00 00 |........?.@.....
The 16-bit word at 0x0e is the reserved sector count, as detailed in Wikipedia. If it isn’t as shown above, SystemACE won’t boot. Unfortunately, recent version of mkdosfs has a new “feature” which silently rounds up the number of reserved sectors to align with clusters. So it gets wrong. The solution for this is to downgrade this simple utility, possibly by downloading it from here. Version 3.0.9 is too new, 2.11 is fine.
Minimalistic setting
If there’s no xilinx.sys file in the root directory, and there is a file with an .ace extension, System ACE will boot from that file. Make sure there’s only one file with the .ace extension in flash’ the root directory. This setting doesn’t take advantage of the possibility to configure which image to boot from at powerup, but it’s easy to start off with.
Configurable setting
We shall now look on a setting which has only one .ace image to boot from, but is easily expanded to several images, chosen by the levels of three pins of the System ACE chip at powerup.
In the root directory, there should be a xilinx.sys file, saying something like this:
# Any comment goes here
dir = trydir;
cfgaddr0 = cfg0;
cfgaddr1 = cfg0;
cfgaddr2 = cfg0;
cfgaddr3 = cfg0;
cfgaddr4 = cfg0;
cfgaddr5 = cfg0;
cfgaddr6 = cfg0;
cfgaddr7 = cfg0;
The eight different cfgaddr lines tell the (Xilinx) System ACE chip which directory to go to, depending on the state of the three CFGADDR pins of the chip. So different profiles can be chosen from with DIP switches and such. In the case above, all eight configuration point at the same directory, cfg0.
The first line, declares the main working directory, which is trydir.
So in the case above, the root directory must have a directory called trydir, and within that directory, there must be a directory called cfg0.
And in cfg0, there must be a single file with .ace suffix, which is the ACE file to be loaded into the FPGA. Or more precisely, the ACE file is a translation of an SVF file, which is a sequence of JTAG instructions.
In order to allow configuration at powerup, create other directories (cfg1, cfg2 etc) and assign them to the desired cfgaddrN in the xilinx.sys file.
Generating the ACE file
Everything said here is related to the software arriving with ISE 13.2. It looks like there have been some significant changes from past versions.
In the Xilinx Platform Studio (EDK), pick Hardware > Generate bitstream on the processor configured. Basically, this generates netlists, builds them, and run the map, place and route and bitgen which creates a file such as system.bit.
Export the hardware format to SDK (Project > Export hardware design to SDK…), and then develop with SDK based upon that hardware. The bundle includes a hardware description as an XML file as well as the bitfile.
Once the project is built, it generates an .elf file, usually in the Debug subfolder. Its name and path is easily found in the Executable tab at the bottom of the SDK. Back in the EDK, pick Project > Select ELF file… and choose the relevant executable (for implementation). Then pick Device Configuration > Update Bitstream. That creates download.bit. This step is mandatory every time the ELF is changed, even though things will most likely work even without updating download.bit every time, since the relevant parts stay the same.
Create a directory to gather the relevant files, and copy the following into it:
- The Tcl script generating ACE file: ISE_DS/EDK/data/xmd/genace.tcl (relative to the path where Xilinx ISE is installed)
- The bitstream (download.bit) file
- The ELF file
Open a command shell (Project > Launch Xilinx Shell if you like), change to this directory and go:
xmd -tcl genace.tcl -hw download.bit -elf myelf.elf -ace myace.ace -board sp605 -target mdm
which generates a lot of junk files (.svf most notably, which contain JTAG commands in a portable format), and eventually the myace.ace is created (any file name is OK, of course).
In the example above, I assumed that the target is the SP605 board. Looking at the genace.tcl script reveals easily which boards are supported. If it isn’t, it’s not such a big deal. The only reason the board matters is because the System ACE needs to know which device in the JTAG chain to talk with plus some programming parameters. The -board flags to this scrips allows setting the options in a “genace option file” (whatever that means). I would hack the script, though. It looks easier. See here for more information.
A test run
At times, the SP605 board’s green LED went on, but nothing happened. Pressing SYS_ACE_RESET is pressed (the middle button out of three close to the Compact Flash jack) caused a reload, which was OK. Probably some kind of race condition during powerup.
References
The walkthrough above is based upon this somewhat outdated guide. The BIST sources (rdf0032.zip) are indeed recommended for download, because of other issues of interest:
- The ready_for_download subdirectory, which shows another example of a Compact Flash layout
- The bootloader/src subdirectory, which has sources for loading executables from the Flash’ filesystem in SREC format (using sysace_fopen and the like).
- The genace_all.sh file in the ready_for_download subdirectory, showing how to create SREC files from ELFs with mb-objcopy.
