Making a custom ISO image.
It turns out it’s no so difficult to make a custom LiveCD / LiveDVD / LiveUSB, even if that includes making small hacks in the target system’s outline. Just follow this guide.
Creating the liveUSB
Create a liveUSB from a liveDVD image with liveusb-creator. Download the ISO image, don’t let liveUSB do it for you. LiveUSB Creator fails more than it succeeds, sometimes because the USB stick isn’t all that reliable, and sometimes it just fails with a yucky Python error message. I wonder if people who choose Python tend to write scripts that break all the time, or if it’s the language which was designed to kick your butt all the time.
For some details and manual generation how-to, see Fedora’s page on this.
The chosen device should be a partition (e.g. /dev/sdd1) and not a drive (e.g. /dev/sdd).
First steps after booting
It looks like a parody on Windows, the way it was 10 years ago: Ugly GUI, a lot of self-importance, and it hangs every now and then.
Let’s face it: Fedora 15 sucks. In retrospective, Fedora 14 is much better, as a LiveUSB and in general.
The persistent image isn’t mounted, so all changes are lost on reboot. There’s a workaround which requires manual intervention on each reboot. The nice thing about this workaround is that I finally found out how to get a boot prompt during boot (useful when exotic installs):
Pressing [Tab] twice on GRUB’s splash, adding the boot parameter
rd.break=pre-trigger
(press ENTER once, and then wait patiently for some 20 seconds). Then at shell prompt
mkdir /overlayfs ; exit
and type “exit” again on the second prompt.
The image comes with a regular user named “liveuser” with an unknown (?) password, and a root account with no password (!). First thing is to open a terminal window (at the bottom of applications. Command line in Linux? Who’s doing that??). Actions:
- Change the passwords
- Open the Firewall GUI and allow SSH. Or just go “service iptables stop” at root prompt if you like to live on the edge.
- Start the sshd service (service sshd start and chkconfig)
- Connect from remote computer.
- Switch to runlevel 3 (“telinit 3″). Installing gcc with graphic display on makes the system hang for whatever reason. But hey, who needs gcc? It’s not like we’re running on a free software system of something.
- Install gcc: “yum install gcc” does the job (one could expect that kernel-devel would require this, but no).
- Install kernel headers: “
yum install kernel-devel“. NOT! That installed the headers for the latest kernel out there, which wasn’t the one I was running. Upgrade the kernel? Really. I wonder if anything of that is left when I reboot.
- To install the headers for the running kernel: “yum install kernel-devel-`uname -r`”
- At which point I compiled a kernel module successfully
Installing to hard disk
# yum install anaconda
# /usr/bin/liveinst
Fooling around
While trying to find my way to making modifications on the LiveCD image, I looked around a bit inside the booted system. The rest of this post is just some probably useless findings. This guide explains how to make the modifications easily.
So, after booting Fedora 15′s liveUSB, I created an mnt directory and went (as root):
# mount -o loop /run/initramfs/live/LiveOS/squashfs.img mnt
mount: warning: mnt seems to be mounted read-only.
The squashfs contains a single file, which is an image of a fullblown ext3 filesystem. It’s a 4 GB file (squashed into 552 MB in the squashfs). So I copied it to a disk, and then mounted it:
# mount -o loop ext3fs.img mnt
Looking around it was evident that there are no traces of any real users in /etc/passwd (not even root), and that there were no subdirectories in /home. The latter issue is easily explained by the following line in the output of plain “mount” on the live system:
/dev/mapper/live-rw on /home type ext4 (rw,noatime,seclabel,barrier=1,data=ordered)
The entire setup is a bit complicated, but it’s pretty easy to find out who the players are:
# dmsetup status
live-osimg-min: 0 8388608 snapshot 2416/2416 24
live-rw: 0 8388608 snapshot 1094288/2129920 4272
# losetup -a
/dev/loop0: [0001]:1898 (/osmin.img)
/dev/loop1: [0700]:2 (/run/initramfs/squashfs.osmin/osmin)
/dev/loop2: [0801]:4 (/run/initramfs/live/LiveOS/squashfs.img)
/dev/loop3: [0702]:3 (/run/initramfs/squashfs/LiveOS/ext3fs.img)
/dev/loop4: [0801]:5 (/overlayfs/LiveOS/overlay-disk-47BA-F8F8)
/dev/loop5: [0801]:4 (/run/initramfs/live/LiveOS/squashfs.img)
This is just a collection of jots I ( = a Drupal newbie) wrote down as I set up a site with Drupal 7.2. Don’t expect this to be more coherent than a typical shopping list.
March 2019 update: Drupal was a huge mistake. Just in case someone out there still has a chance to escape.
And now I appreciate how easy it is to install and use WordPress. Drupal is definitely not good for my health: Hours of frustration trying to accomplish seemingly simple things can’t do me good. It’s tempting to conclude, that Drupal contributors have some job-security-by-obscurity kind of mind set. And I do understand why they end up with nasty hacks in PHP: The nice menus offer a million features, except what is really needed.
General remarks
- The development module is a blessing. In particular the ability to display the SQL queries related to a page: Under Modules, go to “configure” for the Development Module and check “Display query log”. And there’s an “A” next to each row of SQL, which means “display argument”. That is, show the query with what was inside the placeholders. Soooo nice.
- The “devel” tab (part of the development module?) is great: The metadata, with internal names is all listed there. Everything there is to know about a node.
- Enable the PHP text format. It allows running PHP code within any node, which is sometimes necessary.
- The number of files necessary to display a simple page is absolutely insane: A simple textual home page reaches 46 files (many of which are Javascript and CSS files) with a total of 205 kB (!). No wonder the site is slow.
- Take a look on the module list which is sorted by number of installations by default. The things you want are most likely in the beginning of this list.
- Use URL aliases for every post. A node number says nothing and it’s a clear loss of Google juice.
- CKEditor allows the insertion of a DIV container, for which, among others, the language direction can be chosen. This is a great thing for Hebrew, Arabic and other right-to-left languages.
- Drupal’s taxonomy is hierarchical, so the site’s hierarchy can be implemented by setting up a hierarchy of terms within a certain vocabulary, and then tag nodes (pages) in order to place them in the menu hierarchy.
- Flush the entire cache every now and then when changing things.
- The administrative menu gets less quirky as time goes by. Go figure.
- Structure > Blocks is where blocks are added to or removed from the site globally.
- Special pages are best done as Panels. Note that a panel page is just like a normal page with all the blocks around it.
- Special note to self: I’ve hacked modules/system/html.tpl.php (see below). A Drupal upgrade will require rehacking.
Milestones in setting up the site:
- Enable clean URLs
- Enable book module
- Installing a rich text editor (duh…). I went for the classic WYSIWYG module (see below)
- Remove the “powered by Drupal” footer (with all due respect)
- Themes go to tar -xzf in sites/all/themes. Switch themes through Appearance (not any submenu)
- Downloaded and installed (tar -xzf in sites/all/modules) pathauto, Views, Taxonomy Menu, (
CCK is part of Drupal 7), Administration Menu, IMCE (image uploader, I’ll do that manually), Backup and Migrate, and Panels. Ctools was installed as well for Panels. Same goes with Token for pathauto. And then devel. And Disable messages. (Page Title was downloaded but not used to, since it’s great for everything except what I needed: Set the title from a within a view. It’s maybe possible, but I couldn’t figure out how). Views PHP (dev version!) for setting up page title (see below).
- And then enabled the modules, of course.
- Set up cron (run once an hour). I got a lot of “access denied” in the beginning, and then I realized that I should get the link with the cron key under Reports > Status Reports. And then I found out that cron.php denies access when in maintenance mode. So start the cron thing only when the site goes online.
Another thing about cron is that my hosting provider seems to block Curl from running cron jobs with a silly check on the user agent. My updated cron job command hence goes: curl –silent –compressed -A ‘My cronjob’ ‘http://example.com/cron.php?cron_key=(the key supplied on the status page)’
- Set the default page: Configuration > System > Site Information
- Enable statistics: Modules > List > Statistics > Configure. Set to never delete log info (I don’t expect massive traffic).
- Disable Taxonomy term pages for users (see below). Same with node-by-number.
- Ask web host to increase RAM limit (they gave me 128MB, which is OK)
- Enable the “Disable messages” module and configure it to display error messages to admins only (why isn’t this in the core?). This is done by setting permissions (the existence of the module creates new relevant entries). There is no need to configure the module further.
- Edit public_html/sites/all/modules/ctools/includes/cleanstring.inc and replace “\x{d800}” with “\x{e000}” in the expression for CTOOLS_PREG_CLASS_ALNUM. 0xd800 is apparently not a legal Unicode point, and without this change there’s a warning saying exactly that: “Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /home/theuser/public_html/sites/all/modules/ctools/includes/cleanstring.inc).”
- Edit sites/all/themes/arthemia/template.php and change arthemia_primary() at the beginning to read
function arthemia_primary() {
$output = '<div id="page-bar">';
$the_menu_tree = menu_tree(variable_get('menu_main_links_source','main-menu'));
$output .= drupal_render($the_menu_tree);
$output .= '</div>';
return $output;
}
Originally, drupal_render was called with the return value of menu_tree(), an a strict warning is issued about calling a function expecting a reference with a non-reference. Or something.
- April 2018 update: Patch Drupal against this arbitrary code execution exploit (which, BTW, was relevant for current Drupal versions as well. Just being up to date wouldn’t have helped against this one).
Installing the WYSIWYG module
Spoiler: The clear winners are CKEditor and NicEdit. CKEditor because it has the nicest features, but for some reason it has its own spell checked which contacts a dictionary over the web rather than using the browser’s spell checking. And hence it’s disabled by default. It doesn’t make sense to tell the world what you’re typing as a default setting.
- Downloaded from here.
- tar -xzf at sites/all/modules/.
- Created a “libraries” directory under sites/all (why wasn’t it already there?)
- Downloaded CKEditor and did tar -xzf at the “libraries” just created. Be sure that CKEditor wasn’t released after Drupal, because it may not install, and the message “The version of CKEditor could not be detected” will appear at “Wysiwyg profiles”
- Downloaded NicEdit (default configuration), created a “nicedit” directory under “libraries” and unzipped the file there.
- Downloaded YUI, and unzipped it directly on “libraries”
- On web interface menu: Configuration > Text Formats > Add Text format. Add a new text format, e.g. NicEdit and set it for use only for admins (for example).
- Then on web interface menu: Configuration > Wysiwyg profiles assign NicEdit text format to NicEdit editor
- Then on web interface menu: Configuration > Wysiwyg profiles > edit NicEdit > Buttons and plugins and check all checkboxes (TAB and space bar came handy). Or the rich edit box looks like just a plain text box. Why this extra step was necessary is pretty much beyond me.
- Same procedure for other editors.
Setting up a taxonomy menu structure
The sequence below is based upon the Administration Menu module (things may be found in other places using the core interface):
- Create a new menu: Structure > Menus > Add Menu
- Create a new vocabulary: Structure > Taxonomy > Add Vocabulary.
- Generate a few terms in the new vocabulary
- Edit the new vocabulary. Under “Taxonomy menu” pick the the new menu, and check the “Select to rebuild the menu on submit” checkbox. This copies the taxonomy items into the menu (not very sophisticated, is it).
- Now on Structure > Blocks > { your chosen theme } put the menu somewhere. Or more precisely, find the new menu in the list, and assign it a point of appearance. Don’t forget to submit the form.
- To use the menu items as top-page tabs, go to Structure > Menus > Settings and set the source as the new menu
Note that new vocabulary items don’t appear automatically on the menu, and it’s not a matter of flushing the cache or running cron. The menus need to be updated by rebuilding them as mentioned above.
So the bottom line is that I tend to trash the Taxonomy menu and use the book structure.
Tagging pages
- Go to Structure > Content Type > { The content type to alter } > Manage Fields
- Add a new field of type “Term Reference” and select any of the widgets offered.
- Click Save settings
- On the next page, you’ll be asked to choose the vocabulary to use. Save this as well.
- On the next page set up the number of entries allowed (unlimited?) and default tag.
- You may want to hide this field in the display (if it’s for internal uses): Structure > Content Types > { The content type to alter } > Manage Display and set both label and format to hidden. Be sure to apply this to all view modes, full content in particular.
Custom styling for panes
When the GUI tools don’t do the job, editing the theme’s main CSS file does (or a local.css file in the theme directory, but yet another CSS file?). Then, in the content editing page where the panes appear, pick the inner gear menu > CSS Properties and enter a new class name, say mypane-class. In the CSS file setting the attributes for a header within the pane will look like
.mypane-class h2 {
font: Arial, Helvetica, sans-serif;
font-weight: bold;
font-size: 14px;
}
Allowing injected Javascript
Full HTML actually allows Javascript, only it wraps it with CDATA. Which could have been forgiveable, had it done that correctly, and not killed the script on some browsers. The really annoying thing about this is that the purpose of this CDATA statement is merely to silence XHTML validators.
The trick is simple, and explained in a video (I prefer things written down, but anyhow). The idea is to create a new input format. The way Drupal works, it filters the content of the database during page display (the input goes in as is), so the idea is to create a new filter which does nothing. So what is in the database is shown. Not the safest thing in the world for general practice, but sometimes there’s no choice.
Basically, go to the modules list, find “Filter” and click its “configure”. Pick “add text format”, name it “Unfiltered” (or something), make sure none of the filters is checked and that only the administrator has access to this format. That’s it. Now whatever goes in, goes out. Use this format, enter HTML with Javascript and whatever.
Removing the navigation links at the bottom of a book page
As suggested here, don’t edit the original PHP code generating the links, but rather copy /public_html/modules/book/book-navigation.tpl.php to the theme’s directory, edit the copy, and flush the caches.
To retain the links in the lower hierarchy, but remove the navigation links, simply change
<?php if ($has_links): ?>
to
<?php if (0): /* ($has_links): */ ?>
(deleting code is for barbarians)
Disallow access to taxonomy pages
There may be an elegant way to do this with Drupal’s own interface, but I’m yet to discover it. So the trick is to create a custom page, and make it empty. Silly, but it works.
Under Views, clone the view which emulates the core’s taxonomy display (it’s name is simply “Taxonomy”). Add a header, saying whatever you want, which is active even when no items match (who cares). This is merely so you know it’s not the default page you’re watching.
Then set an access rule, saying only admins are allowed: Access > Role > Check administrator.
To get the same effect with nodes, do something similar for node/%. Don’t clone the taxonomy view, since they display taxonomy, and sometimes a page needs to be displayed by node (for admin purposes).
Setting the page title hack
Goal: Set up the page title (the one within <title> tags) based upon the title of a content entry within a view.
I should mention, that what I present here is considered very wrong, in particular because it involves hacking a Drupal core PHP file. But after trying to reach an elegant solution and asking for help, I gave up for something that is ugly but works.
This is the ugly part: modify modules/system/html.tpl.php so that the part saying
<title><?php print $head_title; ?></title>
becomes
<title><?php if (isset($GLOBALS['head_title_override'])) { print $GLOBALS['head_title_override'] ; } else { print $head_title; } ?></title>
While I am ashamed of hacking something under “modules/system”, the usage of PHP global variables may be adequate here, even though I’m sure there are more Drupalish ways to accomplish the same. The advantage of a global variable is that it’s easy to set from any execution context.
Having installed and enabled the “Views PHP” module, there’s now a Global: PHP field. It allows access to the data of fields previously fetched, so having the “title” field already in the list, it’s just
$GLOBALS['head_title_override'] = $row->title;
in the “Value code” window. Note that the available variables are listed just below the window. And this should be excluded from display, of course.
Careful with the PHP: If it’s syntactically wrong, the views edit page may be impossible to reach, so the only way to get out is deleting the view. Disabling auto preview may reduce the risk for this.
Breadcrumbs on Arthemia
For a reason which was beyond me, no breadcrumbs appeared in on my pages. And I ran Arthemia. Until I found the solution here. There’s a bug in Arthemia’s template.php, so
function arthemia_breadcrumb($breadcrumb) {
if (count($breadcrumb) > 1) {
return '<div>'. implode(' › ', $breadcrumb) .'</div>';
}
}
should be replaced with
function arthemia_breadcrumb($breadcrumb) {
if (count($breadcrumb["breadcrumb"]) > 1) {
return '<div>'. implode(' › ', $breadcrumb["breadcrumb"]) .'</div>';
}
}
Looks like the calling convention changed on Drupal 7, but nobody paid attention… This page says a few words about changes in Drupal 7.
Making a local mirror of a site
First, make a copy of the files, of course. Then make a local copy by using a mysqldump backup. Basically, generate the data base with
CREATE DATABASE delme_mirror
from mysql prompt, and then from shell:
mysql -D delme_mirror < databasedump
This can take a few minutes…
Now, allow write-enable on sites/default, make a copy of settings.php within this directory, and make it writable.
Then change the database setting (maybe copy the remote site’s?) so it says something like
$databases = array (
'default' =>
array (
'default' =>
array (
'database' => 'delme_xillybus',
'username' => 'username',
'password' => 'password',
'host' => 'localhost',
'port' => '',
'driver' => 'mysql',
'prefix' => '',
),
),
);
Well, I think this is the thing to do. I didn’t manage to run this, because Drupal went “Fatal error: Undefined class constant ‘MYSQL_ATTR_USE_BUFFERED_QUERY’” which appears to be a PHP version issue (only cutting edge PHP for Drupal, it seems).
Themes
These are my very shallow impressions of a few selected themes I checked up for a hi-tec site. I checked with Firefox 3.6 and IE6 to get a wide range of compatibility issues.
- 0-point: Beautiful and sleek on Firefox, complete disaster on IE6.
- Aqua Marina: Not bad, and survived IE6 test fairly OK, but still pretty bad. Doesn’t have a hi-tec look, though.
- Danblog: Simple & to-the-face design pretty suitable for a hi-tec site, but looks a bit too simple maybe. Survived the IE6 test very well (no differences).
- Danland: Same impression as Danblog. The differences are possibly deeper in.
- Fusion was obviously not intended to be used out of the box.
- Garland looks great, and not so bad on IE6 (resize issues on the latter). Not really a hi-tec look (top region should be bright)
Views
I wrote a separate post on this.
SQL command for strings search in all pages
This is in particular useful for updating links.
It boils down to this:
SELECT entity_id, alias FROM field_data_body LEFT JOIN url_alias ON source=CONCAT('node/', entity_id) WHERE body_value LIKE '%string-to-search%' ORDER BY entity_id;
And then access the page with the URL like e.g. https://thesite.com/node/114 or use the alias as the path instead (if such exists).
Surprisingly enough, I haven’t found a plain and simple cookbook on how to generate a presynthesized IP core for delivery. So I tried it out, and wrote my findings here. Since it’s a result of trial and error, I may have missed some crucial points. Please comment below if you know about such.
Synthesis
In GUI, uncheck “Add I/O Buffers”, set number of clock buffers to zero and set “Pack I/O Registers into IOBS” to No (see below what happens if we don’t). Using these settings, synthesize a very simple module such as this one:
module thecore
(
input clk,
output reg data
);
always @(posedge clk)
data <= !data;
endmodule
and then, on command prompt do
$ ngc2edif thecore.ngc
which will generate an EDIF file as thecore.ndf. Verify by reading this file than nothing else than simple logic elements are implemented. No BUFG or OBUF, for example. These indicate that the synthesizer generated elements which are beyond a core’s scope.
The relevant command-line flags (usually set up in e.g. trycore.xst) are
-iobuf NO -bufg 0 -iob false
Delivery: making an empty HDL file
This is somewhat obvious, and still: The delivery of a binary core consists of an NGC file (or EDIF) and an empty HDL file (headers only). In our simple case this is just
module thecore
(
input clk,
output reg data
);
endmodule
and that’s it. The receiver of the core should be notified to put the NGC file in a special directory. Putting it in the synthesis directory (the one to which the bitfile and other binaries are written) will in general work, but “Cleanup Project Files” does delete it every now and then on ISE 9.2 and possibly other versions as well (not consistently, which makes it even more annoying). The core directory should then be added to “Macro search path” in Translate’s properties, or an -sd “/path/to/mycores” flag in command line use.
If ISE is used (as opposed to command-line build), the NGC file should not be added to the project (ISE will reject it anyhow). The emtpy HDL file should be included in the project, of course.
Is NGC portable?
Can an NGC file by one version of XST be used with an earlier or later design toolset? Can an NGC file targeted for a certain Xilinx device be used for another?
I haven’t found any decisive answer for these questions, which are pretty crucial for whoever wants to release a core in binary format. If anyone has a pointer to a credible statement, please comment below.
What I do know for sure, is that the target device is noted in the NGC file, since that part number appears in the EDIF file generated from it.
I’ve tried to implement very simple designs targeting different devices with the core included (core generated for Spartan 3E, used implementing for Spartan-2 and Virtex-4), and it looks like ngdbuild (which is the one cooking the NGC soup) ignores conflicts in device names. It just runs through.
My educated guess is therefore that the NGC really is only a netlist + constraints, so as long as the logic primitives mentioned there make sense on the target device, nobody’s complaining. Personally, I will push my luck only to the extent of using the NGC within the same device family.
If we do it wrong…
I wanted to see what happens if the synthesis is done without the flags I mentioned above for adjustment. The result is I/O buffers and clock buffers inserted, and ngdbuild will complain with something like
ERROR:NgdBuild:770 - IBUF 'clk_IBUF' and IBUFG 'IBUFG' on net 'clk_IBUF' are
lined up in series. Buffers of the same direction cannot be placed in series.
WARNING:NgdBuild:463 - input pad net 'clk_IBUF' has an illegal input buffer
ERROR:NgdBuild:925 - input net 'clk_IBUF' is connected to the incorrect side of
buffer(s):
pin O on block clk_IBUF with type IBUF
ERROR:NgdBuild:462 - output pad net 'data_OBUF' drives multiple buffers:
pin I on block data_OBUF with type OBUF,
pin I on block ndata1_INV_0 with type INV
ERROR:NgdBuild:467 - output pad net 'data_OBUF' has an illegal buffer
Can I use EDIF?
… or: Can I use ngc2edif’s output as a precompiled core? The truth is that there is no apparent reason to do so, except that some of us prefer knowing what we’re submitting, and a textual file is more helpful in this respect.
The .ndf file generated by ngc2edif was completely ignored by ngdbuild. Renaming its extension to .edf yielded the following “naughty boy” error:
Reading NGO file "L:/delme/tryblackbox/use/usetop.ngc" ...
Executing edif2ngd -noa "mycores\thecore.edf" "_ngo\thecore.ngo"
Release 9.2.03i - edif2ngd J.39
Copyright (c) 1995-2007 Xilinx, Inc. All rights reserved.
INFO:NgdBuild - Release 9.2.03i edif2ngd J.39
INFO:NgdBuild - Copyright (c) 1995-2007 Xilinx, Inc. All rights reserved.
ERROR:NgdBuild:766 - The EDIF netlist 'mycores/thecore.edf' was created by the
Xilinx NGC2EDIF program and is not a valid input netlist. Note that this
EDIF netlist is intended for communicating timing information to third-party
synthesis tools. Specifically, no user modifications to the contents of this
file will effect the final implementation of the design.
ERROR:NgdBuild:276 - edif2ngd exited with errors (return code 1).
Which, simply put says: That EDIF was generated for simulation and not for implementation.
And here’s the nice thing about textual files: They are easy to edit. I mean, how did edif2ngd know who generated the file? Because it starts with
(edif thecore
(edifVersion 2 0 0)
(edifLevel 0)
(keywordMap (keywordLevel 0))
(status
(written
(timestamp 2011 6 16 21 43 12)
(program "Xilinx ngc2edif" (version "J.39"))
(author "Xilinx. Inc ")
(comment "This EDIF netlist is to be used within supported synthesis tools")
(comment "for determining resource/timing estimates of the design component")
(comment "represented by this netlist.")
(comment "Command line: thecore.ngc ")))
So I deleted those two lines having the word “Xilinx”. Now who’s smart?
And indeed, after these lines were deleted, the implementation ran all the way through. Bitfile loaded, FPGA did what I expected it to do. I tested this on a serious project, not this miniature example.
I should mention that I did get some scary warnings by ngc2edit saying “Signal bus X on block Y is not reconstructed, because there are some missing bus signals”. But in the end all is well.
Let’s give Xilinx the credit that they had a good reason to put this little stick in the wheels of using ngc2edif’s output for implementation. That even though it worked for me, it may not work in other situations. Either way, this is not in their standard flow, so I would use EDIFs with caution.
Introduction
I use USB flash sticks for backing up my system periodically by creating an image of the filesystem, and raw-writing it directly to e.g. /dev/sdd1. It’s just a matter of time before I wipe my hard disk by selecting the wrong /dev/something. Or just some other USB stick that happened to be plugged in.
To avoid this, I wrote a udev rule for detecting the two specific USB devices that I use for backing up, and change their ownership, so I won’t be root while doing this (this protects the real disks somewhat). Also, rename the device file to something that can’t be confused with anything else. Ehm, it’s 2018, and systemd’s udev machinery don’t agree renaming /dev files. So add a symbolic link instead. It’s as good, as long as I don’t have to write /dev/sdd-something in the command wiping whatever’s there.
Writing udev rules is well-documented all over the web, but I still wrote down my own quick summary, so I have it handy for the next time. Well, actually twice. The first time was in 2011, and then in 2018, both are on this page. So this is two posts in one.
On systemd-enabled system (that’s all of them by 2018?), the system’s udev rules reside in /lib/udev/. The rules in /etc/udev take precedence, and it’s still the correct place to put local rules. Just don’t be surprised that it’s empty.
And I have another post with udev hacking (and one related to NICs). But now, let’s start with the 2018 post. Updated to Linux Mint 19, with systemd all over the place (and I love it, frankly).
And a little tip to myself and others: man udev. Really. Give it a go every now and then.
Obtaining matching parameters
The built-in udev rules in /lib/udev/rules.d set up a lot of environment variables, which are, among others, printed by e.g.
# udevadm info -q all -n /dev/sdd1
So don’t. It’s better to stick to the actual attributes, as given by
# udevadm info -a -n /dev/sdd1
Note that each segment of this output is a separate device, and hence a separate udev event. All matches in the udev rule must relate to one segment only.
Building the rule
The udevadm info command outputs a very important comment, which is worth repeating:
A rule to match can be composed by the attributes of the device and the attributes from one single parent device.
Based upon the info obtained with udevadm info (see just below), the following rule was set up:
KERNEL=="sd?1", SUBSYSTEM=="block", ATTRS{idProduct}=="5591", ATTRS{idVendor}=="0781", ATTRS{serial}=="1234567890", OWNER:="eli", SYMLINK+="backupstick_a"
One may ask why I bothered to add the match against KERNEL==”sd?1″ and SUBSYSTEM==”block”.
Without these, the match works against e.g. /dev/sdd and /dev/sdd1, even though the symbolic link ends up pointing at /dev/sdd1. However the ownership is set for both, clearly indicating that both sdd and sdd1 were processed. Having some trust in udev, I would expect it to behave in a consistent manner on this, but I don’t want to rely on it. Hence the rule is specific on a first partition (I could have gone ATTR{partition}==”1″ as well, but why).
So the rule above takes advantage of the comment in bold above: Matching from a device and its parent. And it works.
Testing the rules
Note that “udevadm test” performs the actions for real. It’s not a dry run.
There no need for any reload command for testing the rule as follows:
# udevadm test -a add $(udevadm info -q path /dev/sdd) 2>&1 | less
But in order to get it working, a reload is required:
# udevadm control --reload
Picking the correct subsystem
The SUBSYSTEM match determines which device file the rule will apply to. For example, a USB fax/modem device appears both under the “usb” subsystem, which relates to the /dev/bus/usb/XXX/XXX device, and the “tty” subsystem, in which case it goes with the e.g. /dev/ttyACM0 device.
Likewise, when testing, the rule will take effect on one of the device files, and not the other.
The SYMLINK+= command can be useful to tell what device file is related to, but if the symlink is created, odds are we got it right anyhow…
udevadm info of this case
# udevadm info -a -n /dev/sdd1
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd/sdd1':
KERNEL=="sdd1"
SUBSYSTEM=="block"
DRIVER==""
ATTR{alignment_offset}=="0"
ATTR{discard_alignment}=="0"
ATTR{inflight}==" 0 0"
ATTR{partition}=="1"
ATTR{ro}=="0"
ATTR{size}=="242614240"
ATTR{start}=="32"
ATTR{stat}==" 131 0 6184 308 0 0 0 0 0 204 308"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd':
KERNELS=="sdd"
SUBSYSTEMS=="block"
DRIVERS==""
ATTRS{alignment_offset}=="0"
ATTRS{capability}=="51"
ATTRS{discard_alignment}=="0"
ATTRS{events}=="media_change"
ATTRS{events_async}==""
ATTRS{events_poll_msecs}=="-1"
ATTRS{ext_range}=="256"
ATTRS{hidden}=="0"
ATTRS{inflight}==" 0 0"
ATTRS{range}=="16"
ATTRS{removable}=="1"
ATTRS{ro}=="0"
ATTRS{size}=="242614272"
ATTRS{stat}==" 145 0 6296 340 0 0 0 0 0 228 340"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0':
KERNELS=="8:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{blacklist}==""
ATTRS{device_blocked}=="0"
ATTRS{device_busy}=="0"
ATTRS{dh_state}=="detached"
ATTRS{eh_timeout}=="10"
ATTRS{evt_capacity_change_reported}=="0"
ATTRS{evt_inquiry_change_reported}=="0"
ATTRS{evt_lun_change_reported}=="0"
ATTRS{evt_media_change}=="0"
ATTRS{evt_mode_parameter_change_reported}=="0"
ATTRS{evt_soft_threshold_reached}=="0"
ATTRS{inquiry}==""
ATTRS{iocounterbits}=="32"
ATTRS{iodone_cnt}=="0x2ac"
ATTRS{ioerr_cnt}=="0x1"
ATTRS{iorequest_cnt}=="0x2ac"
ATTRS{max_sectors}=="240"
ATTRS{model}=="Ultra USB 3.0 "
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{rev}=="1.00"
ATTRS{scsi_level}=="7"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{type}=="0"
ATTRS{vendor}=="SanDisk "
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0':
KERNELS=="target8:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8':
KERNELS=="host8"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0':
KERNELS=="1-2:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="usb-storage"
ATTRS{authorized}=="1"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bInterfaceClass}=="08"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bInterfaceProtocol}=="50"
ATTRS{bInterfaceSubClass}=="06"
ATTRS{bNumEndpoints}=="02"
ATTRS{supports_autosuspend}=="1"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2':
KERNELS=="1-2"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="224mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0100"
ATTRS{bmAttributes}=="80"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="10"
ATTRS{devpath}=="2"
ATTRS{idProduct}=="5591"
ATTRS{idVendor}=="0781"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="SanDisk"
ATTRS{maxchild}=="0"
ATTRS{product}=="Ultra USB 3.0"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="removable"
ATTRS{serial}=="1234567890"
ATTRS{speed}=="480"
ATTRS{urbnum}=="1580"
ATTRS{version}==" 2.10"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
KERNELS=="usb1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{authorized_default}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="01"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="0mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0415"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{idProduct}=="0002"
ATTRS{idVendor}=="1d6b"
ATTRS{interface_authorized_default}=="1"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Linux 4.15.0-20-generic xhci-hcd"
ATTRS{maxchild}=="16"
ATTRS{product}=="xHCI Host Controller"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{serial}=="0000:00:14.0"
ATTRS{speed}=="480"
ATTRS{urbnum}=="134"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:14.0':
KERNELS=="0000:00:14.0"
SUBSYSTEMS=="pci"
DRIVERS=="xhci_hcd"
ATTRS{broken_parity_status}=="0"
ATTRS{class}=="0x0c0330"
ATTRS{consistent_dma_mask_bits}=="64"
ATTRS{d3cold_allowed}=="1"
ATTRS{dbc}=="disabled"
ATTRS{device}=="0xa2af"
ATTRS{dma_mask_bits}=="64"
ATTRS{driver_override}=="(null)"
ATTRS{enable}=="1"
ATTRS{irq}=="35"
ATTRS{local_cpulist}=="0-11"
ATTRS{local_cpus}=="0,00000000,00000fff"
ATTRS{msi_bus}=="1"
ATTRS{numa_node}=="0"
ATTRS{revision}=="0x00"
ATTRS{subsystem_device}=="0x5007"
ATTRS{subsystem_vendor}=="0x1458"
ATTRS{vendor}=="0x8086"
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
The post from 2011 starts here
Following this excellent guide, I plugged in the USB stick and went
$ udevadm info -a -n /dev/sdd
Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the udev rules key format. A rule to match, can be composed by the attributes of the device and the attributes from one single parent device.
On another computer a similar command could be
$ udevinfo -q all -n /dev/sda
The output from the first command was:
looking at device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0/block/sdd':
KERNEL=="sdd"
SUBSYSTEM=="block"
DRIVER==""
ATTR{range}=="16"
ATTR{ext_range}=="256"
ATTR{removable}=="1"
ATTR{ro}=="0"
ATTR{size}=="7821312"
ATTR{alignment_offset}=="0"
ATTR{discard_alignment}=="0"
ATTR{capability}=="51"
ATTR{stat}==" 49 438 3182 274 0 0 0 0 0 127 274"
ATTR{inflight}==" 0 0"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0':
KERNELS=="15:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{device_blocked}=="0"
ATTRS{type}=="0"
ATTRS{scsi_level}=="3"
ATTRS{vendor}=="SanDisk "
ATTRS{model}=="Cruzer Blade "
ATTRS{rev}=="1.01"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{iocounterbits}=="32"
ATTRS{iorequest_cnt}=="0x1a8"
ATTRS{iodone_cnt}=="0x1a8"
ATTRS{ioerr_cnt}=="0x1"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{evt_media_change}=="0"
ATTRS{dh_state}=="detached"
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{max_sectors}=="240"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0':
KERNELS=="target15:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15':
KERNELS=="host15"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0':
KERNELS=="2-1:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="usb-storage"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bNumEndpoints}=="02"
ATTRS{bInterfaceClass}=="08"
ATTRS{bInterfaceSubClass}=="06"
ATTRS{bInterfaceProtocol}=="50"
ATTRS{modalias}=="usb:v0781p5567d0100dc00dsc00dp00ic08isc06ip50"
ATTRS{supports_autosuspend}=="0"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1':
KERNELS=="2-1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{configuration}==""
ATTRS{bNumInterfaces}==" 1"
ATTRS{bConfigurationValue}=="1"
ATTRS{bmAttributes}=="80"
ATTRS{bMaxPower}=="200mA"
ATTRS{urbnum}=="938"
ATTRS{idVendor}=="0781"
ATTRS{idProduct}=="5567"
ATTRS{bcdDevice}=="0100"
ATTRS{bDeviceClass}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bNumConfigurations}=="1"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{speed}=="480"
ATTRS{busnum}=="2"
ATTRS{devnum}=="8"
ATTRS{devpath}=="1"
ATTRS{version}==" 2.00"
ATTRS{maxchild}=="0"
ATTRS{quirks}=="0x0"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{authorized}=="1"
ATTRS{manufacturer}=="SanDisk"
ATTRS{product}=="Cruzer Blade"
ATTRS{serial}=="200610803009F0712206"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2':
KERNELS=="usb2"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{configuration}==""
ATTRS{bNumInterfaces}==" 1"
ATTRS{bConfigurationValue}=="1"
ATTRS{bmAttributes}=="e0"
ATTRS{bMaxPower}==" 0mA"
ATTRS{urbnum}=="4053"
ATTRS{idVendor}=="1d6b"
ATTRS{idProduct}=="0002"
ATTRS{bcdDevice}=="0206"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bNumConfigurations}=="1"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{speed}=="480"
ATTRS{busnum}=="2"
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{version}==" 2.00"
ATTRS{maxchild}=="8"
ATTRS{quirks}=="0x0"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{authorized}=="1"
ATTRS{manufacturer}=="Linux 2.6.35.4-OCHO1 ehci_hcd"
ATTRS{product}=="EHCI Host Controller"
ATTRS{serial}=="0000:00:1d.7"
ATTRS{authorized_default}=="1"
looking at parent device '/devices/pci0000:00/0000:00:1d.7':
KERNELS=="0000:00:1d.7"
SUBSYSTEMS=="pci"
DRIVERS=="ehci_hcd"
ATTRS{vendor}=="0x8086"
ATTRS{device}=="0x3b34"
ATTRS{subsystem_vendor}=="0x1458"
ATTRS{subsystem_device}=="0x5006"
ATTRS{class}=="0x0c0320"
ATTRS{irq}=="23"
ATTRS{local_cpus}=="00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff"
ATTRS{local_cpulist}=="0-7"
ATTRS{modalias}=="pci:v00008086d00003B34sv00001458sd00005006bc0Csc03i20"
ATTRS{numa_node}=="-1"
ATTRS{dma_mask_bits}=="32"
ATTRS{consistent_dma_mask_bits}=="32"
ATTRS{broken_parity_status}=="0"
ATTRS{msi_bus}==""
ATTRS{companion}==""
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
From which I extracted the idVendor, idProduct and most important, serial attributes, marked in red above.
So I generated /etc/udev/rules.d/10-local-usbstick-rules saying:
SUBSYSTEMS=="usb", ATTRS{idVendor}=="0781", ATTRS{idProduct}=="5567", ATTRS{serial}=="200610803009F0712206", NAME="usbstick_trials", OWNER="eli"
Note that I set myself as the owner of the device file. This means that the data on the USB stick is not so safe, because I can accidentally write data directly to the device file. The upside is that root privileges are not necessary, so the chances for messing up a real hard disk are significantly diminished.
That’s it. /dev/sdd doesn’t appear now, and neither do partition device files, so it’s a bit more difficult to mount the disk (but there’s a trick). On the other hand, raw writing to it just became much safer.
Plug out, plug in again and we have:
$ ls -l /dev/usbstick_trials
brw-rw----. 1 eli disk 8, 48 2011-06-16 17:54 /dev/usbstick_trials
On the other computer (Centos 5.5) the rule was /etc/udev/rules.d/90-local-imagedisk.rules as follows:
KERNEL=="sda1", ENV{ID_SERIAL}=="SATA_WDC_WD5000AAKX-_WD-WCAYUFH69712", NAME="playdisk", OWNER="eli", MODE="666", OPTIONS="last_rule"
The reason for placing the rule late is to allow 50-udev.rules to set up the ID_SERIAL environment variable, which is created from the disk’s serial number. So no other disk gets messed up this way accidentally
Alternative: Monitor events
# udevadm monitor --udev --property
This prints out the udev events as they happen. The relevant info is there too. Useful when you’re not sure which device belongs to what, but a plug-in event makes it very clear. Unfortunately, it doesn’t print out any actions udev takes as it applies the rules.
This is just a raw dump of PCIe communication. I wrote a small sniffer on an FPGA and ran some data in a loop to and from the peripheral. The sniffer’s own data was stored while sniffing, so it doesn’t appear in the stream. The whole thing ran on a Linux machine.
I thought that after writing a few words about TLP formation, a real-life example could be in place.
I recorded headers only, and then hacked together a Perl script (too ugly and specific for any future use) and got the dump below.
All writes from host to peripheral (marked with “>>”) are register writes (to the kernel code the BAR is at 0xfacf2000, but see lspci output below).
Writes from peripheral to host (marked with “<<”) consist of DMA transmissions containing data (longer writes) and status updates (shorter).
And then we have DMA reads made by peripheral, with read requests (“<<”) and completions (“>>”).
Each TLP is given in cleartext, and then the packet’s 3-4 header words hexadecimal in parentheses. In the cleartext part the address and (sender’s) bus ID are given in hex, all other in plain decimal.
As it turned out, the host sends packets using 32-bit addressing, and the peripheral uses 64 bits (as it was told to). Note that the peripheral breaks the standard (section 2.2.4.1) by using 64-bit addressing for addresses that fit 32 bit. Even though it works on most platforms, this is really not recommended.
So before getting to the raw dumps, let’s just see what lspci -vv gave us on the specific device:
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
Subsystem: Xilinx Corporation Generic FPGA core
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step
ping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 4 bytes
Interrupt: pin ? routed to IRQ 42
Region 0: Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 4152
Capabilities: [58] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-0
And now to the dump itself (unfortunately, I didn’t grab any MSI):
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17000
<< (60000020, 010000ff, 00000000, 00c17000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17080
<< (60000020, 010000ff, 00000000, 00c17080)
<< (Write) Type = 0, fmt=3, length=11
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17100
<< (6000000b, 010000ff, 00000000, 00c17100)
<< (Write) Type = 0, fmt=3, length=4
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000004, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15000
<< (60000020, 010000ff, 00000000, 00c15000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15080
<< (60000020, 010000ff, 00000000, 00c15080)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15100
<< (60000020, 010000ff, 00000000, 00c15100)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15180
<< (60000020, 010000ff, 00000000, 00c15180)
<< (Write) Type = 0, fmt=3, length=12
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15200
<< (6000000c, 010000ff, 00000000, 00c15200)
<< (Write) Type = 0, fmt=3, length=4
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000004, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=01
<< Address = 0000000000c1f000
<< (20000080, 010001ff, 00000000, 00c1f000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000140)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000140)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1b000
<< (60000020, 010000ff, 00000000, 00c1b000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000140)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1b080
<< (60000010, 010000ff, 00000000, 00c1b080)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000140)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c000
<< (60000020, 010000ff, 00000000, 00c1c000)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=02
<< Address = 0000000000c1f200
<< (20000080, 010002ff, 00000000, 00c1f200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1f400
<< (60000020, 010000ff, 00000000, 00c1f400)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c100
<< (60000010, 010000ff, 00000000, 00c1c100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000200)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c140
<< (60000020, 010000ff, 00000000, 00c1c140)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000200)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c1c0
<< (60000010, 010000ff, 00000000, 00c1c1c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c200
<< (60000020, 010000ff, 00000000, 00c1c200)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000240)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c280
<< (60000010, 010000ff, 00000000, 00c1c280)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c2c0
<< (60000020, 010000ff, 00000000, 00c1c2c0)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=03
<< Address = 0000000000c20000
<< (20000080, 010003ff, 00000000, 00c20000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000300)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c340
<< (60000020, 010000ff, 00000000, 00c1c340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000340)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c3c0
<< (60000020, 010000ff, 00000000, 00c1c3c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000340)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c440
<< (60000010, 010000ff, 00000000, 00c1c440)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=04
<< Address = 0000000000c20200
<< (20000080, 010004ff, 00000000, 00c20200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c480
<< (60000020, 010000ff, 00000000, 00c1c480)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c500
<< (60000010, 010000ff, 00000000, 00c1c500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000400)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c540
<< (60000020, 010000ff, 00000000, 00c1c540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c5c0
<< (60000020, 010000ff, 00000000, 00c1c5c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000400)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000440)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c640
<< (60000020, 010000ff, 00000000, 00c1c640)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c6c0
<< (60000010, 010000ff, 00000000, 00c1c6c0)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c700
<< (60000010, 010000ff, 00000000, 00c1c700)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=05
<< Address = 0000000000c21000
<< (20000080, 010005ff, 00000000, 00c21000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000500)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c740
<< (60000020, 010000ff, 00000000, 00c1c740)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000540)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c7c0
<< (60000020, 010000ff, 00000000, 00c1c7c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000540)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c840
<< (60000010, 010000ff, 00000000, 00c1c840)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=06
<< Address = 0000000000c21200
<< (20000080, 010006ff, 00000000, 00c21200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c880
<< (60000020, 010000ff, 00000000, 00c1c880)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c900
<< (60000010, 010000ff, 00000000, 00c1c900)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000600)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c940
<< (60000020, 010000ff, 00000000, 00c1c940)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c9c0
<< (60000020, 010000ff, 00000000, 00c1c9c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000600)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000640)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1ca40
<< (60000020, 010000ff, 00000000, 00c1ca40)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cac0
<< (60000010, 010000ff, 00000000, 00c1cac0)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cb00
<< (60000010, 010000ff, 00000000, 00c1cb00)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=07
<< Address = 0000000000c22000
<< (20000080, 010007ff, 00000000, 00c22000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000740)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000740)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cb40
<< (60000020, 010000ff, 00000000, 00c1cb40)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000740)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cbc0
<< (60000010, 010000ff, 00000000, 00c1cbc0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000740)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cc00
<< (60000020, 010000ff, 00000000, 00c1cc00)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c22200
<< (20000080, 010000ff, 00000000, 00c22200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c22400
<< (60000020, 010000ff, 00000000, 00c22400)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cd00
<< (60000010, 010000ff, 00000000, 00c1cd00)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=00
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=00
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000040)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13000
<< (60000020, 010000ff, 00000000, 00c13000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13080
<< (60000020, 010000ff, 00000000, 00c13080)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13100
<< (60000020, 010000ff, 00000000, 00c13100)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13180
<< (60000020, 010000ff, 00000000, 00c13180)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13200
<< (60000020, 010000ff, 00000000, 00c13200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13280
<< (60000020, 010000ff, 00000000, 00c13280)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13300
<< (60000020, 010000ff, 00000000, 00c13300)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13380
<< (60000020, 010000ff, 00000000, 00c13380)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13400
<< (60000020, 010000ff, 00000000, 00c13400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13480
<< (60000020, 010000ff, 00000000, 00c13480)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13500
<< (60000020, 010000ff, 00000000, 00c13500)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13580
<< (60000020, 010000ff, 00000000, 00c13580)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13600
<< (60000020, 010000ff, 00000000, 00c13600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13680
<< (60000020, 010000ff, 00000000, 00c13680)
(and this is where the sniffer's memory got full)
I know that most character drivers don’t implement the flush() method, and the more I deal with it, the more I understand why. But unfortunately, in certain modes of operation flushing is necessary, so I implemented it.
One of the problems with flush() is that it’s called when the file is closed automatically as the process exits. If the device can’t take the remaining data, flushing is stuck, and the process remains in the process table in interruptible sleep mode (which is how I chose to wait in the driver). But in reality, attempting to kill the process results in no response whatsoever, and there’s another interesting thing about this process: Its /proc/fd is empty! So it’s a kind-of zombie, only it isn’t marked as such. But in essence it’s a process waiting for someone to take some data from it, so it can rest in peace.
The only way out is a timeout. The waiting should be something like this:
if (wait_event_interruptible_timeout(dev->wait,
(!dev->rd_full),
HZ) == 0) { /* 1 second timeout */
printk(KERN_WARNING "mydriver: Timed out while flushing. Output data was lost.\n");
mutex_unlock(&dev->wr_mutex);
return -ETIMEDOUT;
}
(Relevant Linux kernel: vanilla 2.6.37)
This really is a note to future self. All of the sudden, terminal windows on my Fedora 12 ceased to respect num lock, shift and CTRL modifiers, and I somehow figured out that it may have something to do with the fact that I was having a Vmplayer machine running.
It’s noted a lot elsewhere, but I’ll write it down for myself here: To get things back to normal, just type
$ setxkbmap
at shell prompt, and all is fine again. This issue is too rare to waste the time to go to the bottom with.
So I bought a cheapo USB modem/fax on Ebay. I have to admit, that I didn’t really expect it to work with my Fedora 12 Linux machine, but it was actually detected as ACM modem, and /dev/ttyACM0 appeared. lsusb -v reveals it’s identified as ID 0572:1329 Conexant Systems (Rockwell), Inc.
Note to self: This is the chosen fax/modem, attached to the computer. As a matter of fact, I have two of these, since I tried to buy another type, but ended up with the same thing exactly.
The modem sends faxes properly and receives them properly when Bezeq dials me to deliver the fax. The command used below is OK with the cellular fax. Just change “filenameprefix” with whatever prefix is desired, and it works slowly but OK.
This tech note shows the fax session sequence, for information.
After symlinking /dev/modem to /dev/ttyACM0, I managed to get “fax send” to work properly. “fax receive” was a different story.
The logs revealed that the problematic part was
efax: 31:04 received 10 bytes: ff c8 c1 00 62 0f 01 00 ef 1c
efax: 31:04 received DCS - session format
efax: 31:04 session 196lpi 9600bps 8.5"/215mm 11"/A4 1D - - 0ms
efax: 31:04 command "+FTS=1"
efax: 31:04 waiting 3.0 s
efax: 31:04 .253 [<CR><LF>OK<CR><LF>]
efax: 31:04 response "OK"
efax: 31:04 command "+FRM=96"
efax: 31:04 waiting 6.0 s
efax: 31:10 Error: timed out after command: +FRM=96
efax: 31:10 command "+FTH=3"
efax: 31:10 waiting 3.1
Which is contrary to my previous thought that a lot of “received UNKNOWN” messages was the problem. Even if these messages have something to do with the real problem, they were still there when I got the fax to work OK.
These commands are well-documented in this document, where I found out that the command stands for receiving data in 9600 bps with no specification of training length. What happened was that something got messed up in the negotiation, leaving the phone line dry, and a timeout soon to come.
The immediate solution was to call efax directly with the -c flag set to reduce the maximal bps rate to 4800. This is painfully slow, but faxes came in OK this way:
/usr/bin/efax -r filenameprefix -c 1,1,0,2,0,0,0,0 -v ewin -v =chewmainrxtf -d/dev/modem -iZ 'i&FE0&D2S7=120' '-i&C0' -iM1L0 -l "+0 000" -kZ
This -c flag is a list of capability flags, as specified in the “man efax” manpage. The second digit, “1″, stands for maximum 4800 bps.
The weird thing is the following part in the (failed) log:
efax: 30:52 using CX93001-EIS_V0.2002-V92 in class 1
efax: 30:52 command "+FRM=?"
efax: 30:52 waiting 5.0 s
efax: 30:52 .437 [<CR><LF>3,24,48,72,73,74,96,97,98,121,122,145,146<CR><LF>]
efax: 30:52 .437 [<CR><LF>OK<CR><LF>]
efax: 30:52 response "OK"
What happens here is that the computer asks the modem for its allowed speeds, and it explicitly says it can take FRM=96 (among others). Annoying.
Anyhow, I think I’ll settle for 4800 bps for the few times I’ll ever need to receive a fax in the future. Solving this simply isn’t worth the effort.
Bonus: send command
For an extreme case with a horrible line (remote responded with EOP to an EOP from my fax, instead of an MCF, which means page OK), I opted for a 4800 bps transmission:
# /usr/bin/efax -v ewin -v chewmainrxtf -d/dev/modem -c 1,1,0,2,0,0,0,0 -x /var/lock/LCK..modem -iZ '-i&FE0&D2S7=120' '-i&C0' -iM1L0 -l '+0 000 000 0000' -kZ -t T077777777 fax-to-send.001
Replace the argument of -T with the number to dial, and the last argument with the file to send.
The header of the fax went out with a UNIX timestamp, since I didn’t state one in the command. But who cares.
While the PCI Express standard is impressive in that it actually makes sense (well, most of the time) there is a pretty annoying thing about read requests reordering.
By the way, I talk about TLP packet formation in general in another post.
In section 2.4.1 of the PCIe spec 1.1, it says that read requests may be reordered as they travel across the switching network, and same goes for read completions associated with different read requests. The read-related packets, which must arrive in the same order they were sent, are read completions associated with the same read request, or as the spec puts it: “… must return in address order”.
This would be a good time to mention, that a read request may be larger than the maximal payload size, so obviously the completer must have a means of splitting the completion into several TLPs, which must be sent in rising address order. And if we’re at it, there’s a boundary restriction on how to cut the data in pieces, namely that the cuts are on boundaries of RCB, where RCB can either be 64 bytes or 128 bytes (if you’re not a Root Complex you may cut on 128-byte boundaries only, and if you are a Root Complex, you choose 64 or 128, and configure the endpoints telling then what you chose).
And there’s a restriction on the maximal read request size, but that’s a different story.
So far the specification makes sense: Read completions will be split into several TLPs pretty often, and they have to arrive to the requester in linear address order, so these packets must not be reordered.
But what if the endpoint needs to collect a chunk of data which is larger than the maximal read request size (typically 512 bytes)? It will have to issue several read requests for that. But read requests and completions from different read requests may be reordered.
So if we want to assure that the data arrives in linear order (which is necessary when the data goes into a FIFO-like data sink) each read request can be transmitted only when the last completion TLP from the previous request arrives. Otherwise, a completion TLP from the following request may arrive before that last packet. Hence there’s a time gap of non-utilized bandwidth.
In general there is no problem having several outstanding read requests. So had read requests and read completions been strictly ordered, it would be possible to send the following read request more or less immediately after the first one, and completions would arrive continuously.
Another issue, which is less likely to bother anyone, is that if some software makes assumptions on the order at which data in some buffer is updated, this can cause rare bugs. For example, if the read completions update some dual port RAM, and there’s software reading from this buffer, it may deduce from the update of some high-addressed memory cell that the entire buffer is updated.
And a final word: Since PCIe infrastructure is pretty plain when this post is written, I will be surprised if anyone manages to catch any packet reordering taking place for real. But exactly as write reordering is commonplace in modern CPUs to increase performance, it’s only a matter of time before PCIe switching networks do the same.
Update: I got an email from someone informing me that he spotted reordering taking place on some Intel desktop chipsets. So this isn’t just theory, after all.
Questions & Comments
Since the comment section of similar posts tends to turn into a Q & A session, I’ve taken that to a separate place. So if you’d like to discuss something with me, please post questions and comments here instead. No registration is required. The comment section below is closed.
Posted Under:
PCI express
This post was written by
eli on May 24, 2011
Comments Off
Finding the maximal payload manually
The truth is, there is no need to do this manually. lspci does the work for us. But looking into the configuration table once and for all helps demystifying the issue. So here we go.
According to the PCIe spec (section 7.8), the max_payload_size the card can take is give in the PCIe Device Capabilities Register (Offset 0x04 in the PCI Express Capability structure), bits 2-0. Basically, take that three-bit field as a number, add 7 to it, and you have the log-2 of the number of bytes allowed.
Let me write this in C for clarity:
max_payload_size_capable = 1 << ( (DevCapReg & 0x07) + 7); // In bytes
The actual value used is set by host in the Device Control Register (Offset Ox08 in the PCI Express Capability structure). It’s the same drill, but with bits 7-5 instead. So in C it would be
max_payload_size_in_effect = 1 << ( ( (DevCtrlReg >> 5) & 0x07) + 7); // In bytes
OK, so how can we find these registers? How do we find the structure? Let’s start with dumping the hexadecimal representation of the 256-byte configuration space. Using lspci -xxx on a Linux machine we will get the dump for all devices, but we’ll look at one specific:
# lspci -xxx
(...)
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
00: ee 10 34 12 07 04 10 00 00 00 00 ff 01 00 00 00
10: 04 f0 af fd 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 ee 10 34 12
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 00 00 00
40: 01 48 03 70 08 00 00 00 05 58 81 00 0c 30 e0 fe
50: 00 00 00 00 71 41 00 00 10 00 01 00 c2 8f 28 00
60: 10 28 00 00 11 f4 03 00 00 00 11 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The first important thing to know about lspci -xxx on a little-endian machine (x86 processors included) is that PCI and PCIe work in big endian. And that the data is shown as little-endian DWs (or 32-bit unsigned ints). So the way to look at the output is in groups of four bytes each, and take them for a little-endian unsigned int, whose bit map matches the spec.
For example, according to the spec, bits 15-0 of the word mapped at 00h is the Vendor ID, and bits 31-16 is the Device ID. So we take the first four bytes for a little-endian 32-bit integer, and get Ox123410ee. Bits 15-0 are indeed Ox10ee, the vendor ID Xilinx, and bits 31-16 are Ox1234 which is the Device ID I made up for a custom device. So far so good.
Now we need to find the PCI Express Capability structure. It’s one of the structures in a linked list (would you believe that?), and it’s identified by a Cap ID of Ox10.
The pointer to the list is at bits 7-0 of the configuration word at Ox34. In our little-endian representation above, it’s simply the byte at Ox34, which says Ox40. The capabilities hence start at Ox40.
From here on, we can travel along the list of capability structures. Each starts 32-bit aligned, with the header always having the Capability ID on bits 7-0 (appears as the first byte above), and a pointer to the next structure in bits 15-8 (the second byte).
So we start at offset Ox40, finding it’s of Cap ID Ox01, and that the byte at offset Ox41 tells us that the next entry is at offset Ox48. Moving on to offset Ox48 we find Cap ID Ox05 and the next entry at Ox58. The entry at Ox58 is with Cap ID Ox10 (!!!), and it’s the last one (pointer to next is zero).
So we found our structure at Ox58. The Device Capabilities Register is hence at Ox5c (offset Ox04) and reads Ox00288fc2. The Device Control Register is at Ox60 (offset Ox08), and reads Ox00002810.
So we learn from bits 2-0 of the Device Capabilities Register (having value 2) that the device supports a max_payload_size of 512. But bits 7-5 (having value 0) of the Device Control Register tell us that the effective maximal payload is only 128 bytes.
Getting the info with lspci
As I mentioned above, we didn’t really need to find the addresses by hand. lspci -v gives us, for the specific device:
# lspci -v
(...)
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
Subsystem: Xilinx Corporation Generic FPGA core
Flags: bus master, fast devsel, latency 0, IRQ 42
Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Capabilities: [58] Express Endpoint IRQ 0
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
So the address to the PCI Express capabilities structure is given to us, but not the internal details (maybe some newer version of lspci does). And by the way, the size=128 above has nothing to do with maximal payload: It’s the size of the memory space allocated to the device by BIOS (BAR address space, if we’re into it).
For the details, including the maximal payload, we use the lspci -vv option.
# lspci -vv
(...)
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
Subsystem: Xilinx Corporation Generic FPGA core
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 4 bytes
Interrupt: pin ? routed to IRQ 42
Region 0: Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 4171
Capabilities: [58] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-0
So there we have it, black on white: The device supports 512 bytes MaxPayload, but below we have MayPayload given as 128 bytes.
Impact on performance
A 128-byte maximal payload is not good news if one wants to get the most out of the bandwidth. By the way, switches are not permitted to split packets (but the Root Complex is allowed) so this number actually tells us how much overhead each TLP (Transaction Layer Packet) carries. I talk about the TLP structure in another post.
Let’s make a quick calculation: Each packet comes with a header of 3 DWs (a DW is a 32-bit word, right?) when using 32 bit addressing, and a header of 4 DWs for 64-bit addressing. Let’s be nice and assume 32-bit addressing, so the header is 3 DWs.
TLPs may optionally carry a one-DW TLP digest (ECRC), which is generally a stupid idea if you trust the switching chipsets not to mess up your data. Otherwise, the Data Link layer’s CRC should be enough. So we’ll assume no TLP digest.
The Data Link layer overhead is a bit more difficult to estimate, because it has its own housekeeping packets. But since most acknowledge and flow control packets go in the opposite direction and hence don’t interfere with a unidirectional bulk data transmission, we’ll focus on the actual data added to each TLP: It consists of a 2-byte header (partially filled with a TLP sequence number) and a 4-byte LCRC.
So all in all, the overhead, assuming a 3-DW header, is 12 bytes for the TLP header and another 6 bytes by the Data Link. All in all, we have 18 bytes, which takes up ~12% if transmitted along a 128-byte TLP, but only ~3.4% for a 512-byte TLP.
For a 1x configuration, which has 2.5 Gbps on the wires, and effective 2.0 Gbps (10/8 bit coding), we could dream about 250 MBytes/sec. But when the TLPs are 128 bytes long each, our upper limit goes down to some ~219 Mbytes/sec. With 512-bytes TLPs it’s ~241 Mbytes/sec. Does it matter at all? I suppose it depends. In benchmark testing, it’s important to know these limits, or you start thinking something is wrong, when it’s actually the packet network limiting the speed.
