The fatal error
Let’s break the bad news: Spartan-6′s PCIe core may drop TLP packets sporadically when ASPM (Active State Power Management) is enabled. That means that any TLP given to the core for transmission can silently disappear, as if it was never submitted. I also suspect that the problem exists in the opposite direction.
Hardware involved: Spartan xc6slx45t-fgg484-3-es (evaluation sample version) on an SP605 evaluation board. That mounted on a Gigabyte G31M-ES2L motherboard, having the Intel G33 chipset and a E5700 3.0 GHz processor.
The fairly good news is that he core’s cfg_dstatus[2] ( = fatal error detected) will go high as a result of dropping TLPs. Or at least so it did in my case. So it looks like monitoring this signal, and do something loud if it goes to ’1′ is enough to at least know if the core does the job or not.
Let me spell it out: If you’re designing with Xilinx’ PCIe core, you should verify that cfg_dstatus[2] stays ’0′, and if it goes high you should treat the PCIe endpoint as completely unreliable.
How to know if ASPM is enabled
On a Linux box, become root and go lspci -vv. The output will include all devices, but the relevant part will be something like
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
Subsystem: Xilinx Corporation Generic FPGA core
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 4 bytes
Interrupt: pin ? routed to IRQ 44
Region 0: Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 4181
Capabilities: [58] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM L0s Enabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
There we have it: I set up the device with an unlimited L0s latency, hence the BIOS configured the device to have an unlimited L0s latency, and this ended up with ASPM enabled.
What we really want is the output to end with something like:
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
The elegant solution
The really good news is that there is a simple solution: Disable ASPM. In other words, program the link partners to never reach the L0s nor L1 power saving modes. In a Linux kernel driver, it’s pretty simple:
#include <linux/pci-aspm.h>
pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 |
PCIE_LINK_STATE_CLKPM)
This is something I would do without thinking twice for any device based upon Xilinx’ PCIe core. Actually, I would do this for any device for which power saving is irrelevant.
The maybe-working solution
In theory, the kernel can run in different ASPM policies, one of which is “powersave”. If it runs in “performance” all transactions to L0s are disabled, and all should be well. In practice, it looks like the kernel community is pushing towards allowing L0s even under the performance policy.
The shaky workaround
When some software wants to allow L0s, it must check if the switching latency from L0s to L0 (that is, from napping to awake) is one the device can take. The device announces its maximal allowed latency in the PCI Express Capability Structure. By setting the acceptable L0s latency limit to the shortest latency allowed (64 ns), one can hope that the hardware will not be able to meet this requirement, and hence give up on using ASPM. This trick happened to work on my own motherboard, but another motherboard may be able to meet the 64 ns requirement, and enable ASPM. So this isn’t really a solution.
Anyhow, the success of this method will yield an lspci -vv output with something like
Capabilities: [58] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
How I know it isn’t my own bug
The transitions from L0 to L0s and back throttle the data flow through the PCIe core, so maybe these on-and-offs exposed a bug in my own HDL code’s data flow? Why do I blame Xilinx?
The answer was found in the dbg_* debug lines supplied from within the PCIe core. These lines go high whenever something bad happens in the core’s lower layers. Running without ASPM these lines stayed zero. When ASPM was enabled, and in conjunction with packet drops, the following lines were asserted:
- dbg_reg_detected_fatal: Well, I knew this already. A fatal error was detected.
- dbg_reg_detected_correctable: A correctable error was detected. Nice, but I really don’t care.
- dbg_rply_timeout_status: The replay timer fired off: A TLP packet was sent, but didn’t receive an acknowledgement. That indicates that things aren’t perfect, but if the packet was retransmitted, this doesn’t indicate a user-visible issue.
- dbg_dl_protocol_status: Ayeee. This means that an out of range ACK or NAK was received. In other words, the link partners are not on the same page regarding which packets are waiting for acknowledgement.
The last bullet is our smoking gun: It indicates that the PCIe link protocol has been violated. There is nothing the application HDL code can do to make this happen. The two last bullets indicate some problem in the domain of a TLP being lost, retransmitted, and some problem with the acknowledge. Not a sign saying “a packet was lost”, but as close as one gets to that, I suppose.
Update: My attention to some interesting Xilinx Answer records was drawn in a comment below. Answer record #33871 mentions LL_REPLAY_TIMEOUT as the a parameter to fix, in order to solve a fatal error condition, but says nothing about packet dropping. It looks like this issue has been fixed in the official PCIe wrapper lately. This leaves me wondering whether people didn’t notice they lost packets, or if Xilinx decided not to admit that too loud.
Making a custom ISO image.
It turns out it’s no so difficult to make a custom LiveCD / LiveDVD / LiveUSB, even if that includes making small hacks in the target system’s outline. Just follow this guide.
Creating the liveUSB
Create a liveUSB from a liveDVD image with liveusb-creator. Download the ISO image, don’t let liveUSB do it for you. LiveUSB Creator fails more than it succeeds, sometimes because the USB stick isn’t all that reliable, and sometimes it just fails with a yucky Python error message. I wonder if people who choose Python tend to write scripts that break all the time, or if it’s the language which was designed to kick your butt all the time.
For some details and manual generation how-to, see Fedora’s page on this.
The chosen device should be a partition (e.g. /dev/sdd1) and not a drive (e.g. /dev/sdd).
First steps after booting
It looks like a parody on Windows, the way it was 10 years ago: Ugly GUI, a lot of self-importance, and it hangs every now and then.
Let’s face it: Fedora 15 sucks. In retrospective, Fedora 14 is much better, as a LiveUSB and in general.
The persistent image isn’t mounted, so all changes are lost on reboot. There’s a workaround which requires manual intervention on each reboot. The nice thing about this workaround is that I finally found out how to get a boot prompt during boot (useful when exotic installs):
Pressing [Tab] twice on GRUB’s splash, adding the boot parameter
rd.break=pre-trigger
(press ENTER once, and then wait patiently for some 20 seconds). Then at shell prompt
mkdir /overlayfs ; exit
and type “exit” again on the second prompt.
The image comes with a regular user named “liveuser” with an unknown (?) password, and a root account with no password (!). First thing is to open a terminal window (at the bottom of applications. Command line in Linux? Who’s doing that??). Actions:
- Change the passwords
- Open the Firewall GUI and allow SSH. Or just go “service iptables stop” at root prompt if you like to live on the edge.
- Start the sshd service (service sshd start and chkconfig)
- Connect from remote computer.
- Switch to runlevel 3 (“telinit 3″). Installing gcc with graphic display on makes the system hang for whatever reason. But hey, who needs gcc? It’s not like we’re running on a free software system of something.
- Install gcc: “yum install gcc” does the job (one could expect that kernel-devel would require this, but no).
- Install kernel headers: “
yum install kernel-devel“. NOT! That installed the headers for the latest kernel out there, which wasn’t the one I was running. Upgrade the kernel? Really. I wonder if anything of that is left when I reboot.
- To install the headers for the running kernel: “yum install kernel-devel-`uname -r`”
- At which point I compiled a kernel module successfully
Installing to hard disk
# yum install anaconda
# /usr/bin/liveinst
Fooling around
While trying to find my way to making modifications on the LiveCD image, I looked around a bit inside the booted system. The rest of this post is just some probably useless findings. This guide explains how to make the modifications easily.
So, after booting Fedora 15′s liveUSB, I created an mnt directory and went (as root):
# mount -o loop /run/initramfs/live/LiveOS/squashfs.img mnt
mount: warning: mnt seems to be mounted read-only.
The squashfs contains a single file, which is an image of a fullblown ext3 filesystem. It’s a 4 GB file (squashed into 552 MB in the squashfs). So I copied it to a disk, and then mounted it:
# mount -o loop ext3fs.img mnt
Looking around it was evident that there are no traces of any real users in /etc/passwd (not even root), and that there were no subdirectories in /home. The latter issue is easily explained by the following line in the output of plain “mount” on the live system:
/dev/mapper/live-rw on /home type ext4 (rw,noatime,seclabel,barrier=1,data=ordered)
The entire setup is a bit complicated, but it’s pretty easy to find out who the players are:
# dmsetup status
live-osimg-min: 0 8388608 snapshot 2416/2416 24
live-rw: 0 8388608 snapshot 1094288/2129920 4272
# losetup -a
/dev/loop0: [0001]:1898 (/osmin.img)
/dev/loop1: [0700]:2 (/run/initramfs/squashfs.osmin/osmin)
/dev/loop2: [0801]:4 (/run/initramfs/live/LiveOS/squashfs.img)
/dev/loop3: [0702]:3 (/run/initramfs/squashfs/LiveOS/ext3fs.img)
/dev/loop4: [0801]:5 (/overlayfs/LiveOS/overlay-disk-47BA-F8F8)
/dev/loop5: [0801]:4 (/run/initramfs/live/LiveOS/squashfs.img)
This is just a collection of jots I ( = a Drupal newbie) wrote down as I set up a site with Drupal 7.2. Don’t expect this to be more coherent than a typical shopping list.
March 2019 update: Drupal was a huge mistake. Just in case someone out there still has a chance to escape.
And now I appreciate how easy it is to install and use WordPress. Drupal is definitely not good for my health: Hours of frustration trying to accomplish seemingly simple things can’t do me good. It’s tempting to conclude, that Drupal contributors have some job-security-by-obscurity kind of mind set. And I do understand why they end up with nasty hacks in PHP: The nice menus offer a million features, except what is really needed.
General remarks
- The development module is a blessing. In particular the ability to display the SQL queries related to a page: Under Modules, go to “configure” for the Development Module and check “Display query log”. And there’s an “A” next to each row of SQL, which means “display argument”. That is, show the query with what was inside the placeholders. Soooo nice.
- The “devel” tab (part of the development module?) is great: The metadata, with internal names is all listed there. Everything there is to know about a node.
- Enable the PHP text format. It allows running PHP code within any node, which is sometimes necessary.
- The number of files necessary to display a simple page is absolutely insane: A simple textual home page reaches 46 files (many of which are Javascript and CSS files) with a total of 205 kB (!). No wonder the site is slow.
- Take a look on the module list which is sorted by number of installations by default. The things you want are most likely in the beginning of this list.
- Use URL aliases for every post. A node number says nothing and it’s a clear loss of Google juice.
- CKEditor allows the insertion of a DIV container, for which, among others, the language direction can be chosen. This is a great thing for Hebrew, Arabic and other right-to-left languages.
- Drupal’s taxonomy is hierarchical, so the site’s hierarchy can be implemented by setting up a hierarchy of terms within a certain vocabulary, and then tag nodes (pages) in order to place them in the menu hierarchy.
- Flush the entire cache every now and then when changing things.
- The administrative menu gets less quirky as time goes by. Go figure.
- Structure > Blocks is where blocks are added to or removed from the site globally.
- Special pages are best done as Panels. Note that a panel page is just like a normal page with all the blocks around it.
- Special note to self: I’ve hacked modules/system/html.tpl.php (see below). A Drupal upgrade will require rehacking.
Milestones in setting up the site:
- Enable clean URLs
- Enable book module
- Installing a rich text editor (duh…). I went for the classic WYSIWYG module (see below)
- Remove the “powered by Drupal” footer (with all due respect)
- Themes go to tar -xzf in sites/all/themes. Switch themes through Appearance (not any submenu)
- Downloaded and installed (tar -xzf in sites/all/modules) pathauto, Views, Taxonomy Menu, (
CCK is part of Drupal 7), Administration Menu, IMCE (image uploader, I’ll do that manually), Backup and Migrate, and Panels. Ctools was installed as well for Panels. Same goes with Token for pathauto. And then devel. And Disable messages. (Page Title was downloaded but not used to, since it’s great for everything except what I needed: Set the title from a within a view. It’s maybe possible, but I couldn’t figure out how). Views PHP (dev version!) for setting up page title (see below).
- And then enabled the modules, of course.
- Set up cron (run once an hour). I got a lot of “access denied” in the beginning, and then I realized that I should get the link with the cron key under Reports > Status Reports. And then I found out that cron.php denies access when in maintenance mode. So start the cron thing only when the site goes online.
Another thing about cron is that my hosting provider seems to block Curl from running cron jobs with a silly check on the user agent. My updated cron job command hence goes: curl –silent –compressed -A ‘My cronjob’ ‘http://example.com/cron.php?cron_key=(the key supplied on the status page)’
- Set the default page: Configuration > System > Site Information
- Enable statistics: Modules > List > Statistics > Configure. Set to never delete log info (I don’t expect massive traffic).
- Disable Taxonomy term pages for users (see below). Same with node-by-number.
- Ask web host to increase RAM limit (they gave me 128MB, which is OK)
- Enable the “Disable messages” module and configure it to display error messages to admins only (why isn’t this in the core?). This is done by setting permissions (the existence of the module creates new relevant entries). There is no need to configure the module further.
- Edit public_html/sites/all/modules/ctools/includes/cleanstring.inc and replace “\x{d800}” with “\x{e000}” in the expression for CTOOLS_PREG_CLASS_ALNUM. 0xd800 is apparently not a legal Unicode point, and without this change there’s a warning saying exactly that: “Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /home/theuser/public_html/sites/all/modules/ctools/includes/cleanstring.inc).”
- Edit sites/all/themes/arthemia/template.php and change arthemia_primary() at the beginning to read
function arthemia_primary() {
$output = '<div id="page-bar">';
$the_menu_tree = menu_tree(variable_get('menu_main_links_source','main-menu'));
$output .= drupal_render($the_menu_tree);
$output .= '</div>';
return $output;
}
Originally, drupal_render was called with the return value of menu_tree(), an a strict warning is issued about calling a function expecting a reference with a non-reference. Or something.
- April 2018 update: Patch Drupal against this arbitrary code execution exploit (which, BTW, was relevant for current Drupal versions as well. Just being up to date wouldn’t have helped against this one).
Installing the WYSIWYG module
Spoiler: The clear winners are CKEditor and NicEdit. CKEditor because it has the nicest features, but for some reason it has its own spell checked which contacts a dictionary over the web rather than using the browser’s spell checking. And hence it’s disabled by default. It doesn’t make sense to tell the world what you’re typing as a default setting.
- Downloaded from here.
- tar -xzf at sites/all/modules/.
- Created a “libraries” directory under sites/all (why wasn’t it already there?)
- Downloaded CKEditor and did tar -xzf at the “libraries” just created. Be sure that CKEditor wasn’t released after Drupal, because it may not install, and the message “The version of CKEditor could not be detected” will appear at “Wysiwyg profiles”
- Downloaded NicEdit (default configuration), created a “nicedit” directory under “libraries” and unzipped the file there.
- Downloaded YUI, and unzipped it directly on “libraries”
- On web interface menu: Configuration > Text Formats > Add Text format. Add a new text format, e.g. NicEdit and set it for use only for admins (for example).
- Then on web interface menu: Configuration > Wysiwyg profiles assign NicEdit text format to NicEdit editor
- Then on web interface menu: Configuration > Wysiwyg profiles > edit NicEdit > Buttons and plugins and check all checkboxes (TAB and space bar came handy). Or the rich edit box looks like just a plain text box. Why this extra step was necessary is pretty much beyond me.
- Same procedure for other editors.
Setting up a taxonomy menu structure
The sequence below is based upon the Administration Menu module (things may be found in other places using the core interface):
- Create a new menu: Structure > Menus > Add Menu
- Create a new vocabulary: Structure > Taxonomy > Add Vocabulary.
- Generate a few terms in the new vocabulary
- Edit the new vocabulary. Under “Taxonomy menu” pick the the new menu, and check the “Select to rebuild the menu on submit” checkbox. This copies the taxonomy items into the menu (not very sophisticated, is it).
- Now on Structure > Blocks > { your chosen theme } put the menu somewhere. Or more precisely, find the new menu in the list, and assign it a point of appearance. Don’t forget to submit the form.
- To use the menu items as top-page tabs, go to Structure > Menus > Settings and set the source as the new menu
Note that new vocabulary items don’t appear automatically on the menu, and it’s not a matter of flushing the cache or running cron. The menus need to be updated by rebuilding them as mentioned above.
So the bottom line is that I tend to trash the Taxonomy menu and use the book structure.
Tagging pages
- Go to Structure > Content Type > { The content type to alter } > Manage Fields
- Add a new field of type “Term Reference” and select any of the widgets offered.
- Click Save settings
- On the next page, you’ll be asked to choose the vocabulary to use. Save this as well.
- On the next page set up the number of entries allowed (unlimited?) and default tag.
- You may want to hide this field in the display (if it’s for internal uses): Structure > Content Types > { The content type to alter } > Manage Display and set both label and format to hidden. Be sure to apply this to all view modes, full content in particular.
Custom styling for panes
When the GUI tools don’t do the job, editing the theme’s main CSS file does (or a local.css file in the theme directory, but yet another CSS file?). Then, in the content editing page where the panes appear, pick the inner gear menu > CSS Properties and enter a new class name, say mypane-class. In the CSS file setting the attributes for a header within the pane will look like
.mypane-class h2 {
font: Arial, Helvetica, sans-serif;
font-weight: bold;
font-size: 14px;
}
Allowing injected Javascript
Full HTML actually allows Javascript, only it wraps it with CDATA. Which could have been forgiveable, had it done that correctly, and not killed the script on some browsers. The really annoying thing about this is that the purpose of this CDATA statement is merely to silence XHTML validators.
The trick is simple, and explained in a video (I prefer things written down, but anyhow). The idea is to create a new input format. The way Drupal works, it filters the content of the database during page display (the input goes in as is), so the idea is to create a new filter which does nothing. So what is in the database is shown. Not the safest thing in the world for general practice, but sometimes there’s no choice.
Basically, go to the modules list, find “Filter” and click its “configure”. Pick “add text format”, name it “Unfiltered” (or something), make sure none of the filters is checked and that only the administrator has access to this format. That’s it. Now whatever goes in, goes out. Use this format, enter HTML with Javascript and whatever.
Removing the navigation links at the bottom of a book page
As suggested here, don’t edit the original PHP code generating the links, but rather copy /public_html/modules/book/book-navigation.tpl.php to the theme’s directory, edit the copy, and flush the caches.
To retain the links in the lower hierarchy, but remove the navigation links, simply change
<?php if ($has_links): ?>
to
<?php if (0): /* ($has_links): */ ?>
(deleting code is for barbarians)
Disallow access to taxonomy pages
There may be an elegant way to do this with Drupal’s own interface, but I’m yet to discover it. So the trick is to create a custom page, and make it empty. Silly, but it works.
Under Views, clone the view which emulates the core’s taxonomy display (it’s name is simply “Taxonomy”). Add a header, saying whatever you want, which is active even when no items match (who cares). This is merely so you know it’s not the default page you’re watching.
Then set an access rule, saying only admins are allowed: Access > Role > Check administrator.
To get the same effect with nodes, do something similar for node/%. Don’t clone the taxonomy view, since they display taxonomy, and sometimes a page needs to be displayed by node (for admin purposes).
Setting the page title hack
Goal: Set up the page title (the one within <title> tags) based upon the title of a content entry within a view.
I should mention, that what I present here is considered very wrong, in particular because it involves hacking a Drupal core PHP file. But after trying to reach an elegant solution and asking for help, I gave up for something that is ugly but works.
This is the ugly part: modify modules/system/html.tpl.php so that the part saying
<title><?php print $head_title; ?></title>
becomes
<title><?php if (isset($GLOBALS['head_title_override'])) { print $GLOBALS['head_title_override'] ; } else { print $head_title; } ?></title>
While I am ashamed of hacking something under “modules/system”, the usage of PHP global variables may be adequate here, even though I’m sure there are more Drupalish ways to accomplish the same. The advantage of a global variable is that it’s easy to set from any execution context.
Having installed and enabled the “Views PHP” module, there’s now a Global: PHP field. It allows access to the data of fields previously fetched, so having the “title” field already in the list, it’s just
$GLOBALS['head_title_override'] = $row->title;
in the “Value code” window. Note that the available variables are listed just below the window. And this should be excluded from display, of course.
Careful with the PHP: If it’s syntactically wrong, the views edit page may be impossible to reach, so the only way to get out is deleting the view. Disabling auto preview may reduce the risk for this.
Breadcrumbs on Arthemia
For a reason which was beyond me, no breadcrumbs appeared in on my pages. And I ran Arthemia. Until I found the solution here. There’s a bug in Arthemia’s template.php, so
function arthemia_breadcrumb($breadcrumb) {
if (count($breadcrumb) > 1) {
return '<div>'. implode(' › ', $breadcrumb) .'</div>';
}
}
should be replaced with
function arthemia_breadcrumb($breadcrumb) {
if (count($breadcrumb["breadcrumb"]) > 1) {
return '<div>'. implode(' › ', $breadcrumb["breadcrumb"]) .'</div>';
}
}
Looks like the calling convention changed on Drupal 7, but nobody paid attention… This page says a few words about changes in Drupal 7.
Making a local mirror of a site
First, make a copy of the files, of course. Then make a local copy by using a mysqldump backup. Basically, generate the data base with
CREATE DATABASE delme_mirror
from mysql prompt, and then from shell:
mysql -D delme_mirror < databasedump
This can take a few minutes…
Now, allow write-enable on sites/default, make a copy of settings.php within this directory, and make it writable.
Then change the database setting (maybe copy the remote site’s?) so it says something like
$databases = array (
'default' =>
array (
'default' =>
array (
'database' => 'delme_xillybus',
'username' => 'username',
'password' => 'password',
'host' => 'localhost',
'port' => '',
'driver' => 'mysql',
'prefix' => '',
),
),
);
Well, I think this is the thing to do. I didn’t manage to run this, because Drupal went “Fatal error: Undefined class constant ‘MYSQL_ATTR_USE_BUFFERED_QUERY’” which appears to be a PHP version issue (only cutting edge PHP for Drupal, it seems).
Themes
These are my very shallow impressions of a few selected themes I checked up for a hi-tec site. I checked with Firefox 3.6 and IE6 to get a wide range of compatibility issues.
- 0-point: Beautiful and sleek on Firefox, complete disaster on IE6.
- Aqua Marina: Not bad, and survived IE6 test fairly OK, but still pretty bad. Doesn’t have a hi-tec look, though.
- Danblog: Simple & to-the-face design pretty suitable for a hi-tec site, but looks a bit too simple maybe. Survived the IE6 test very well (no differences).
- Danland: Same impression as Danblog. The differences are possibly deeper in.
- Fusion was obviously not intended to be used out of the box.
- Garland looks great, and not so bad on IE6 (resize issues on the latter). Not really a hi-tec look (top region should be bright)
Views
I wrote a separate post on this.
SQL command for strings search in all pages
This is in particular useful for updating links.
It boils down to this:
SELECT entity_id, alias FROM field_data_body LEFT JOIN url_alias ON source=CONCAT('node/', entity_id) WHERE body_value LIKE '%string-to-search%' ORDER BY entity_id;
And then access the page with the URL like e.g. https://thesite.com/node/114 or use the alias as the path instead (if such exists).
Surprisingly enough, I haven’t found a plain and simple cookbook on how to generate a presynthesized IP core for delivery. So I tried it out, and wrote my findings here. Since it’s a result of trial and error, I may have missed some crucial points. Please comment below if you know about such.
Synthesis
In GUI, uncheck “Add I/O Buffers”, set number of clock buffers to zero and set “Pack I/O Registers into IOBS” to No (see below what happens if we don’t). Using these settings, synthesize a very simple module such as this one:
module thecore
(
input clk,
output reg data
);
always @(posedge clk)
data <= !data;
endmodule
and then, on command prompt do
$ ngc2edif thecore.ngc
which will generate an EDIF file as thecore.ndf. Verify by reading this file than nothing else than simple logic elements are implemented. No BUFG or OBUF, for example. These indicate that the synthesizer generated elements which are beyond a core’s scope.
The relevant command-line flags (usually set up in e.g. trycore.xst) are
-iobuf NO -bufg 0 -iob false
Delivery: making an empty HDL file
This is somewhat obvious, and still: The delivery of a binary core consists of an NGC file (or EDIF) and an empty HDL file (headers only). In our simple case this is just
module thecore
(
input clk,
output reg data
);
endmodule
and that’s it. The receiver of the core should be notified to put the NGC file in a special directory. Putting it in the synthesis directory (the one to which the bitfile and other binaries are written) will in general work, but “Cleanup Project Files” does delete it every now and then on ISE 9.2 and possibly other versions as well (not consistently, which makes it even more annoying). The core directory should then be added to “Macro search path” in Translate’s properties, or an -sd “/path/to/mycores” flag in command line use.
If ISE is used (as opposed to command-line build), the NGC file should not be added to the project (ISE will reject it anyhow). The emtpy HDL file should be included in the project, of course.
Is NGC portable?
Can an NGC file by one version of XST be used with an earlier or later design toolset? Can an NGC file targeted for a certain Xilinx device be used for another?
I haven’t found any decisive answer for these questions, which are pretty crucial for whoever wants to release a core in binary format. If anyone has a pointer to a credible statement, please comment below.
What I do know for sure, is that the target device is noted in the NGC file, since that part number appears in the EDIF file generated from it.
I’ve tried to implement very simple designs targeting different devices with the core included (core generated for Spartan 3E, used implementing for Spartan-2 and Virtex-4), and it looks like ngdbuild (which is the one cooking the NGC soup) ignores conflicts in device names. It just runs through.
My educated guess is therefore that the NGC really is only a netlist + constraints, so as long as the logic primitives mentioned there make sense on the target device, nobody’s complaining. Personally, I will push my luck only to the extent of using the NGC within the same device family.
If we do it wrong…
I wanted to see what happens if the synthesis is done without the flags I mentioned above for adjustment. The result is I/O buffers and clock buffers inserted, and ngdbuild will complain with something like
ERROR:NgdBuild:770 - IBUF 'clk_IBUF' and IBUFG 'IBUFG' on net 'clk_IBUF' are
lined up in series. Buffers of the same direction cannot be placed in series.
WARNING:NgdBuild:463 - input pad net 'clk_IBUF' has an illegal input buffer
ERROR:NgdBuild:925 - input net 'clk_IBUF' is connected to the incorrect side of
buffer(s):
pin O on block clk_IBUF with type IBUF
ERROR:NgdBuild:462 - output pad net 'data_OBUF' drives multiple buffers:
pin I on block data_OBUF with type OBUF,
pin I on block ndata1_INV_0 with type INV
ERROR:NgdBuild:467 - output pad net 'data_OBUF' has an illegal buffer
Can I use EDIF?
… or: Can I use ngc2edif’s output as a precompiled core? The truth is that there is no apparent reason to do so, except that some of us prefer knowing what we’re submitting, and a textual file is more helpful in this respect.
The .ndf file generated by ngc2edif was completely ignored by ngdbuild. Renaming its extension to .edf yielded the following “naughty boy” error:
Reading NGO file "L:/delme/tryblackbox/use/usetop.ngc" ...
Executing edif2ngd -noa "mycores\thecore.edf" "_ngo\thecore.ngo"
Release 9.2.03i - edif2ngd J.39
Copyright (c) 1995-2007 Xilinx, Inc. All rights reserved.
INFO:NgdBuild - Release 9.2.03i edif2ngd J.39
INFO:NgdBuild - Copyright (c) 1995-2007 Xilinx, Inc. All rights reserved.
ERROR:NgdBuild:766 - The EDIF netlist 'mycores/thecore.edf' was created by the
Xilinx NGC2EDIF program and is not a valid input netlist. Note that this
EDIF netlist is intended for communicating timing information to third-party
synthesis tools. Specifically, no user modifications to the contents of this
file will effect the final implementation of the design.
ERROR:NgdBuild:276 - edif2ngd exited with errors (return code 1).
Which, simply put says: That EDIF was generated for simulation and not for implementation.
And here’s the nice thing about textual files: They are easy to edit. I mean, how did edif2ngd know who generated the file? Because it starts with
(edif thecore
(edifVersion 2 0 0)
(edifLevel 0)
(keywordMap (keywordLevel 0))
(status
(written
(timestamp 2011 6 16 21 43 12)
(program "Xilinx ngc2edif" (version "J.39"))
(author "Xilinx. Inc ")
(comment "This EDIF netlist is to be used within supported synthesis tools")
(comment "for determining resource/timing estimates of the design component")
(comment "represented by this netlist.")
(comment "Command line: thecore.ngc ")))
So I deleted those two lines having the word “Xilinx”. Now who’s smart?
And indeed, after these lines were deleted, the implementation ran all the way through. Bitfile loaded, FPGA did what I expected it to do. I tested this on a serious project, not this miniature example.
I should mention that I did get some scary warnings by ngc2edit saying “Signal bus X on block Y is not reconstructed, because there are some missing bus signals”. But in the end all is well.
Let’s give Xilinx the credit that they had a good reason to put this little stick in the wheels of using ngc2edif’s output for implementation. That even though it worked for me, it may not work in other situations. Either way, this is not in their standard flow, so I would use EDIFs with caution.
Introduction
I use USB flash sticks for backing up my system periodically by creating an image of the filesystem, and raw-writing it directly to e.g. /dev/sdd1. It’s just a matter of time before I wipe my hard disk by selecting the wrong /dev/something. Or just some other USB stick that happened to be plugged in.
To avoid this, I wrote a udev rule for detecting the two specific USB devices that I use for backing up, and change their ownership, so I won’t be root while doing this (this protects the real disks somewhat). Also, rename the device file to something that can’t be confused with anything else. Ehm, it’s 2018, and systemd’s udev machinery don’t agree renaming /dev files. So add a symbolic link instead. It’s as good, as long as I don’t have to write /dev/sdd-something in the command wiping whatever’s there.
Writing udev rules is well-documented all over the web, but I still wrote down my own quick summary, so I have it handy for the next time. Well, actually twice. The first time was in 2011, and then in 2018, both are on this page. So this is two posts in one.
On systemd-enabled system (that’s all of them by 2018?), the system’s udev rules reside in /lib/udev/. The rules in /etc/udev take precedence, and it’s still the correct place to put local rules. Just don’t be surprised that it’s empty.
And I have another post with udev hacking (and one related to NICs). But now, let’s start with the 2018 post. Updated to Linux Mint 19, with systemd all over the place (and I love it, frankly).
And a little tip to myself and others: man udev. Really. Give it a go every now and then.
Obtaining matching parameters
The built-in udev rules in /lib/udev/rules.d set up a lot of environment variables, which are, among others, printed by e.g.
# udevadm info -q all -n /dev/sdd1
So don’t. It’s better to stick to the actual attributes, as given by
# udevadm info -a -n /dev/sdd1
Note that each segment of this output is a separate device, and hence a separate udev event. All matches in the udev rule must relate to one segment only.
Building the rule
The udevadm info command outputs a very important comment, which is worth repeating:
A rule to match can be composed by the attributes of the device and the attributes from one single parent device.
Based upon the info obtained with udevadm info (see just below), the following rule was set up:
KERNEL=="sd?1", SUBSYSTEM=="block", ATTRS{idProduct}=="5591", ATTRS{idVendor}=="0781", ATTRS{serial}=="1234567890", OWNER:="eli", SYMLINK+="backupstick_a"
One may ask why I bothered to add the match against KERNEL==”sd?1″ and SUBSYSTEM==”block”.
Without these, the match works against e.g. /dev/sdd and /dev/sdd1, even though the symbolic link ends up pointing at /dev/sdd1. However the ownership is set for both, clearly indicating that both sdd and sdd1 were processed. Having some trust in udev, I would expect it to behave in a consistent manner on this, but I don’t want to rely on it. Hence the rule is specific on a first partition (I could have gone ATTR{partition}==”1″ as well, but why).
So the rule above takes advantage of the comment in bold above: Matching from a device and its parent. And it works.
Testing the rules
Note that “udevadm test” performs the actions for real. It’s not a dry run.
There no need for any reload command for testing the rule as follows:
# udevadm test -a add $(udevadm info -q path /dev/sdd) 2>&1 | less
But in order to get it working, a reload is required:
# udevadm control --reload
Picking the correct subsystem
The SUBSYSTEM match determines which device file the rule will apply to. For example, a USB fax/modem device appears both under the “usb” subsystem, which relates to the /dev/bus/usb/XXX/XXX device, and the “tty” subsystem, in which case it goes with the e.g. /dev/ttyACM0 device.
Likewise, when testing, the rule will take effect on one of the device files, and not the other.
The SYMLINK+= command can be useful to tell what device file is related to, but if the symlink is created, odds are we got it right anyhow…
udevadm info of this case
# udevadm info -a -n /dev/sdd1
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd/sdd1':
KERNEL=="sdd1"
SUBSYSTEM=="block"
DRIVER==""
ATTR{alignment_offset}=="0"
ATTR{discard_alignment}=="0"
ATTR{inflight}==" 0 0"
ATTR{partition}=="1"
ATTR{ro}=="0"
ATTR{size}=="242614240"
ATTR{start}=="32"
ATTR{stat}==" 131 0 6184 308 0 0 0 0 0 204 308"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0/block/sdd':
KERNELS=="sdd"
SUBSYSTEMS=="block"
DRIVERS==""
ATTRS{alignment_offset}=="0"
ATTRS{capability}=="51"
ATTRS{discard_alignment}=="0"
ATTRS{events}=="media_change"
ATTRS{events_async}==""
ATTRS{events_poll_msecs}=="-1"
ATTRS{ext_range}=="256"
ATTRS{hidden}=="0"
ATTRS{inflight}==" 0 0"
ATTRS{range}=="16"
ATTRS{removable}=="1"
ATTRS{ro}=="0"
ATTRS{size}=="242614272"
ATTRS{stat}==" 145 0 6296 340 0 0 0 0 0 228 340"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0/8:0:0:0':
KERNELS=="8:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{blacklist}==""
ATTRS{device_blocked}=="0"
ATTRS{device_busy}=="0"
ATTRS{dh_state}=="detached"
ATTRS{eh_timeout}=="10"
ATTRS{evt_capacity_change_reported}=="0"
ATTRS{evt_inquiry_change_reported}=="0"
ATTRS{evt_lun_change_reported}=="0"
ATTRS{evt_media_change}=="0"
ATTRS{evt_mode_parameter_change_reported}=="0"
ATTRS{evt_soft_threshold_reached}=="0"
ATTRS{inquiry}==""
ATTRS{iocounterbits}=="32"
ATTRS{iodone_cnt}=="0x2ac"
ATTRS{ioerr_cnt}=="0x1"
ATTRS{iorequest_cnt}=="0x2ac"
ATTRS{max_sectors}=="240"
ATTRS{model}=="Ultra USB 3.0 "
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{rev}=="1.00"
ATTRS{scsi_level}=="7"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{type}=="0"
ATTRS{vendor}=="SanDisk "
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8/target8:0:0':
KERNELS=="target8:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/host8':
KERNELS=="host8"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0':
KERNELS=="1-2:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="usb-storage"
ATTRS{authorized}=="1"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bInterfaceClass}=="08"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bInterfaceProtocol}=="50"
ATTRS{bInterfaceSubClass}=="06"
ATTRS{bNumEndpoints}=="02"
ATTRS{supports_autosuspend}=="1"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-2':
KERNELS=="1-2"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="224mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0100"
ATTRS{bmAttributes}=="80"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="10"
ATTRS{devpath}=="2"
ATTRS{idProduct}=="5591"
ATTRS{idVendor}=="0781"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="SanDisk"
ATTRS{maxchild}=="0"
ATTRS{product}=="Ultra USB 3.0"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="removable"
ATTRS{serial}=="1234567890"
ATTRS{speed}=="480"
ATTRS{urbnum}=="1580"
ATTRS{version}==" 2.10"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
KERNELS=="usb1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{authorized_default}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="01"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="0mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0415"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{idProduct}=="0002"
ATTRS{idVendor}=="1d6b"
ATTRS{interface_authorized_default}=="1"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Linux 4.15.0-20-generic xhci-hcd"
ATTRS{maxchild}=="16"
ATTRS{product}=="xHCI Host Controller"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{serial}=="0000:00:14.0"
ATTRS{speed}=="480"
ATTRS{urbnum}=="134"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:14.0':
KERNELS=="0000:00:14.0"
SUBSYSTEMS=="pci"
DRIVERS=="xhci_hcd"
ATTRS{broken_parity_status}=="0"
ATTRS{class}=="0x0c0330"
ATTRS{consistent_dma_mask_bits}=="64"
ATTRS{d3cold_allowed}=="1"
ATTRS{dbc}=="disabled"
ATTRS{device}=="0xa2af"
ATTRS{dma_mask_bits}=="64"
ATTRS{driver_override}=="(null)"
ATTRS{enable}=="1"
ATTRS{irq}=="35"
ATTRS{local_cpulist}=="0-11"
ATTRS{local_cpus}=="0,00000000,00000fff"
ATTRS{msi_bus}=="1"
ATTRS{numa_node}=="0"
ATTRS{revision}=="0x00"
ATTRS{subsystem_device}=="0x5007"
ATTRS{subsystem_vendor}=="0x1458"
ATTRS{vendor}=="0x8086"
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
The post from 2011 starts here
Following this excellent guide, I plugged in the USB stick and went
$ udevadm info -a -n /dev/sdd
Udevadm info starts with the device specified by the devpath and then walks up the chain of parent devices. It prints for every device found, all possible attributes in the udev rules key format. A rule to match, can be composed by the attributes of the device and the attributes from one single parent device.
On another computer a similar command could be
$ udevinfo -q all -n /dev/sda
The output from the first command was:
looking at device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0/block/sdd':
KERNEL=="sdd"
SUBSYSTEM=="block"
DRIVER==""
ATTR{range}=="16"
ATTR{ext_range}=="256"
ATTR{removable}=="1"
ATTR{ro}=="0"
ATTR{size}=="7821312"
ATTR{alignment_offset}=="0"
ATTR{discard_alignment}=="0"
ATTR{capability}=="51"
ATTR{stat}==" 49 438 3182 274 0 0 0 0 0 127 274"
ATTR{inflight}==" 0 0"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0/15:0:0:0':
KERNELS=="15:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{device_blocked}=="0"
ATTRS{type}=="0"
ATTRS{scsi_level}=="3"
ATTRS{vendor}=="SanDisk "
ATTRS{model}=="Cruzer Blade "
ATTRS{rev}=="1.01"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{iocounterbits}=="32"
ATTRS{iorequest_cnt}=="0x1a8"
ATTRS{iodone_cnt}=="0x1a8"
ATTRS{ioerr_cnt}=="0x1"
ATTRS{modalias}=="scsi:t-0x00"
ATTRS{evt_media_change}=="0"
ATTRS{dh_state}=="detached"
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{max_sectors}=="240"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15/target15:0:0':
KERNELS=="target15:0:0"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/host15':
KERNELS=="host15"
SUBSYSTEMS=="scsi"
DRIVERS==""
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0':
KERNELS=="2-1:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="usb-storage"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bNumEndpoints}=="02"
ATTRS{bInterfaceClass}=="08"
ATTRS{bInterfaceSubClass}=="06"
ATTRS{bInterfaceProtocol}=="50"
ATTRS{modalias}=="usb:v0781p5567d0100dc00dsc00dp00ic08isc06ip50"
ATTRS{supports_autosuspend}=="0"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2/2-1':
KERNELS=="2-1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{configuration}==""
ATTRS{bNumInterfaces}==" 1"
ATTRS{bConfigurationValue}=="1"
ATTRS{bmAttributes}=="80"
ATTRS{bMaxPower}=="200mA"
ATTRS{urbnum}=="938"
ATTRS{idVendor}=="0781"
ATTRS{idProduct}=="5567"
ATTRS{bcdDevice}=="0100"
ATTRS{bDeviceClass}=="00"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bNumConfigurations}=="1"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{speed}=="480"
ATTRS{busnum}=="2"
ATTRS{devnum}=="8"
ATTRS{devpath}=="1"
ATTRS{version}==" 2.00"
ATTRS{maxchild}=="0"
ATTRS{quirks}=="0x0"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{authorized}=="1"
ATTRS{manufacturer}=="SanDisk"
ATTRS{product}=="Cruzer Blade"
ATTRS{serial}=="200610803009F0712206"
looking at parent device '/devices/pci0000:00/0000:00:1d.7/usb2':
KERNELS=="usb2"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{configuration}==""
ATTRS{bNumInterfaces}==" 1"
ATTRS{bConfigurationValue}=="1"
ATTRS{bmAttributes}=="e0"
ATTRS{bMaxPower}==" 0mA"
ATTRS{urbnum}=="4053"
ATTRS{idVendor}=="1d6b"
ATTRS{idProduct}=="0002"
ATTRS{bcdDevice}=="0206"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bDeviceProtocol}=="00"
ATTRS{bNumConfigurations}=="1"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{speed}=="480"
ATTRS{busnum}=="2"
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{version}==" 2.00"
ATTRS{maxchild}=="8"
ATTRS{quirks}=="0x0"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{authorized}=="1"
ATTRS{manufacturer}=="Linux 2.6.35.4-OCHO1 ehci_hcd"
ATTRS{product}=="EHCI Host Controller"
ATTRS{serial}=="0000:00:1d.7"
ATTRS{authorized_default}=="1"
looking at parent device '/devices/pci0000:00/0000:00:1d.7':
KERNELS=="0000:00:1d.7"
SUBSYSTEMS=="pci"
DRIVERS=="ehci_hcd"
ATTRS{vendor}=="0x8086"
ATTRS{device}=="0x3b34"
ATTRS{subsystem_vendor}=="0x1458"
ATTRS{subsystem_device}=="0x5006"
ATTRS{class}=="0x0c0320"
ATTRS{irq}=="23"
ATTRS{local_cpus}=="00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff"
ATTRS{local_cpulist}=="0-7"
ATTRS{modalias}=="pci:v00008086d00003B34sv00001458sd00005006bc0Csc03i20"
ATTRS{numa_node}=="-1"
ATTRS{dma_mask_bits}=="32"
ATTRS{consistent_dma_mask_bits}=="32"
ATTRS{broken_parity_status}=="0"
ATTRS{msi_bus}==""
ATTRS{companion}==""
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
From which I extracted the idVendor, idProduct and most important, serial attributes, marked in red above.
So I generated /etc/udev/rules.d/10-local-usbstick-rules saying:
SUBSYSTEMS=="usb", ATTRS{idVendor}=="0781", ATTRS{idProduct}=="5567", ATTRS{serial}=="200610803009F0712206", NAME="usbstick_trials", OWNER="eli"
Note that I set myself as the owner of the device file. This means that the data on the USB stick is not so safe, because I can accidentally write data directly to the device file. The upside is that root privileges are not necessary, so the chances for messing up a real hard disk are significantly diminished.
That’s it. /dev/sdd doesn’t appear now, and neither do partition device files, so it’s a bit more difficult to mount the disk (but there’s a trick). On the other hand, raw writing to it just became much safer.
Plug out, plug in again and we have:
$ ls -l /dev/usbstick_trials
brw-rw----. 1 eli disk 8, 48 2011-06-16 17:54 /dev/usbstick_trials
On the other computer (Centos 5.5) the rule was /etc/udev/rules.d/90-local-imagedisk.rules as follows:
KERNEL=="sda1", ENV{ID_SERIAL}=="SATA_WDC_WD5000AAKX-_WD-WCAYUFH69712", NAME="playdisk", OWNER="eli", MODE="666", OPTIONS="last_rule"
The reason for placing the rule late is to allow 50-udev.rules to set up the ID_SERIAL environment variable, which is created from the disk’s serial number. So no other disk gets messed up this way accidentally
Alternative: Monitor events
# udevadm monitor --udev --property
This prints out the udev events as they happen. The relevant info is there too. Useful when you’re not sure which device belongs to what, but a plug-in event makes it very clear. Unfortunately, it doesn’t print out any actions udev takes as it applies the rules.
This is just a raw dump of PCIe communication. I wrote a small sniffer on an FPGA and ran some data in a loop to and from the peripheral. The sniffer’s own data was stored while sniffing, so it doesn’t appear in the stream. The whole thing ran on a Linux machine.
I thought that after writing a few words about TLP formation, a real-life example could be in place.
I recorded headers only, and then hacked together a Perl script (too ugly and specific for any future use) and got the dump below.
All writes from host to peripheral (marked with “>>”) are register writes (to the kernel code the BAR is at 0xfacf2000, but see lspci output below).
Writes from peripheral to host (marked with “<<”) consist of DMA transmissions containing data (longer writes) and status updates (shorter).
And then we have DMA reads made by peripheral, with read requests (“<<”) and completions (“>>”).
Each TLP is given in cleartext, and then the packet’s 3-4 header words hexadecimal in parentheses. In the cleartext part the address and (sender’s) bus ID are given in hex, all other in plain decimal.
As it turned out, the host sends packets using 32-bit addressing, and the peripheral uses 64 bits (as it was told to). Note that the peripheral breaks the standard (section 2.2.4.1) by using 64-bit addressing for addresses that fit 32 bit. Even though it works on most platforms, this is really not recommended.
So before getting to the raw dumps, let’s just see what lspci -vv gave us on the specific device:
01:00.0 Class ff00: Xilinx Corporation Generic FPGA core
Subsystem: Xilinx Corporation Generic FPGA core
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step
ping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 4 bytes
Interrupt: pin ? routed to IRQ 42
Region 0: Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 4152
Capabilities: [58] Express Endpoint IRQ 0
Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-0
And now to the dump itself (unfortunately, I didn’t grab any MSI):
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17000
<< (60000020, 010000ff, 00000000, 00c17000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17080
<< (60000020, 010000ff, 00000000, 00c17080)
<< (Write) Type = 0, fmt=3, length=11
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c17100
<< (6000000b, 010000ff, 00000000, 00c17100)
<< (Write) Type = 0, fmt=3, length=4
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000004, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15000
<< (60000020, 010000ff, 00000000, 00c15000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15080
<< (60000020, 010000ff, 00000000, 00c15080)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15100
<< (60000020, 010000ff, 00000000, 00c15100)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15180
<< (60000020, 010000ff, 00000000, 00c15180)
<< (Write) Type = 0, fmt=3, length=12
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c15200
<< (6000000c, 010000ff, 00000000, 00c15200)
<< (Write) Type = 0, fmt=3, length=4
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000004, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=01
<< Address = 0000000000c1f000
<< (20000080, 010001ff, 00000000, 00c1f000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000140)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000140)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1b000
<< (60000020, 010000ff, 00000000, 00c1b000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000140)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1b080
<< (60000010, 010000ff, 00000000, 00c1b080)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=01
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000140)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c000
<< (60000020, 010000ff, 00000000, 00c1c000)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=02
<< Address = 0000000000c1f200
<< (20000080, 010002ff, 00000000, 00c1f200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1f400
<< (60000020, 010000ff, 00000000, 00c1f400)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c100
<< (60000010, 010000ff, 00000000, 00c1c100)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000200)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c140
<< (60000020, 010000ff, 00000000, 00c1c140)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000200)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c1c0
<< (60000010, 010000ff, 00000000, 00c1c1c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000240)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c200
<< (60000020, 010000ff, 00000000, 00c1c200)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=02
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000240)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c280
<< (60000010, 010000ff, 00000000, 00c1c280)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c2c0
<< (60000020, 010000ff, 00000000, 00c1c2c0)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=03
<< Address = 0000000000c20000
<< (20000080, 010003ff, 00000000, 00c20000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000300)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c340
<< (60000020, 010000ff, 00000000, 00c1c340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000340)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000340)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c3c0
<< (60000020, 010000ff, 00000000, 00c1c3c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000300)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=03
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000340)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c440
<< (60000010, 010000ff, 00000000, 00c1c440)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=04
<< Address = 0000000000c20200
<< (20000080, 010004ff, 00000000, 00c20200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c480
<< (60000020, 010000ff, 00000000, 00c1c480)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c500
<< (60000010, 010000ff, 00000000, 00c1c500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000400)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c540
<< (60000020, 010000ff, 00000000, 00c1c540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c5c0
<< (60000020, 010000ff, 00000000, 00c1c5c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000440)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000400)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=04
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000440)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c640
<< (60000020, 010000ff, 00000000, 00c1c640)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c6c0
<< (60000010, 010000ff, 00000000, 00c1c6c0)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c700
<< (60000010, 010000ff, 00000000, 00c1c700)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=05
<< Address = 0000000000c21000
<< (20000080, 010005ff, 00000000, 00c21000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000500)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c740
<< (60000020, 010000ff, 00000000, 00c1c740)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000540)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000540)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c7c0
<< (60000020, 010000ff, 00000000, 00c1c7c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000500)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=05
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000540)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c840
<< (60000010, 010000ff, 00000000, 00c1c840)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=06
<< Address = 0000000000c21200
<< (20000080, 010006ff, 00000000, 00c21200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c880
<< (60000020, 010000ff, 00000000, 00c1c880)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c900
<< (60000010, 010000ff, 00000000, 00c1c900)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000600)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c940
<< (60000020, 010000ff, 00000000, 00c1c940)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1c9c0
<< (60000020, 010000ff, 00000000, 00c1c9c0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000640)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000600)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=06
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000640)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1ca40
<< (60000020, 010000ff, 00000000, 00c1ca40)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cac0
<< (60000010, 010000ff, 00000000, 00c1cac0)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cb00
<< (60000010, 010000ff, 00000000, 00c1cb00)
<< (Write) Type = 0, fmt=3, length=2
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c29200
<< (60000002, 010000ff, 00000000, 00c29200)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff008
>> (40000001, 0000000f, fdaff008)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff034
>> (40000001, 0000000f, fdaff034)
>> (Write) Type = 0, fmt=2, length=1
>> Bus ID: 0000, Tag=00
>> Address = fdaff030
>> (40000001, 0000000f, fdaff030)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=07
<< Address = 0000000000c22000
<< (20000080, 010007ff, 00000000, 00c22000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000740)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=384
>> (4a000010, 00000180, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=320
>> (4a000010, 00000140, 01000740)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cb40
<< (60000020, 010000ff, 00000000, 00c1cb40)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=256
>> (4a000010, 00000100, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=192
>> (4a000010, 000000c0, 01000740)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cbc0
<< (60000010, 010000ff, 00000000, 00c1cbc0)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=0, byte count=128
>> (4a000010, 00000080, 01000700)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=07
>> Completion low addr=64, byte count=64
>> (4a000010, 00000040, 01000740)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cc00
<< (60000020, 010000ff, 00000000, 00c1cc00)
<< (Read Rq) Type = 0, fmt=1, length=128
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c22200
<< (20000080, 010000ff, 00000000, 00c22200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c22400
<< (60000020, 010000ff, 00000000, 00c22400)
<< (Write) Type = 0, fmt=3, length=16
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c1cd00
<< (60000010, 010000ff, 00000000, 00c1cd00)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=00
>> Completion low addr=0, byte count=512
>> (4a000010, 00000200, 01000000)
>> (Completion) Type = 10, fmt=2, length=16
>> Bus ID: 0000, Tag=00
>> Completion low addr=64, byte count=448
>> (4a000010, 000001c0, 01000040)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13000
<< (60000020, 010000ff, 00000000, 00c13000)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13080
<< (60000020, 010000ff, 00000000, 00c13080)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13100
<< (60000020, 010000ff, 00000000, 00c13100)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13180
<< (60000020, 010000ff, 00000000, 00c13180)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13200
<< (60000020, 010000ff, 00000000, 00c13200)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13280
<< (60000020, 010000ff, 00000000, 00c13280)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13300
<< (60000020, 010000ff, 00000000, 00c13300)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13380
<< (60000020, 010000ff, 00000000, 00c13380)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13400
<< (60000020, 010000ff, 00000000, 00c13400)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13480
<< (60000020, 010000ff, 00000000, 00c13480)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13500
<< (60000020, 010000ff, 00000000, 00c13500)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13580
<< (60000020, 010000ff, 00000000, 00c13580)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13600
<< (60000020, 010000ff, 00000000, 00c13600)
<< (Write) Type = 0, fmt=3, length=32
<< Bus ID: 0100, Tag=00
<< Address = 0000000000c13680
<< (60000020, 010000ff, 00000000, 00c13680)
(and this is where the sniffer's memory got full)
I know that most character drivers don’t implement the flush() method, and the more I deal with it, the more I understand why. But unfortunately, in certain modes of operation flushing is necessary, so I implemented it.
One of the problems with flush() is that it’s called when the file is closed automatically as the process exits. If the device can’t take the remaining data, flushing is stuck, and the process remains in the process table in interruptible sleep mode (which is how I chose to wait in the driver). But in reality, attempting to kill the process results in no response whatsoever, and there’s another interesting thing about this process: Its /proc/fd is empty! So it’s a kind-of zombie, only it isn’t marked as such. But in essence it’s a process waiting for someone to take some data from it, so it can rest in peace.
The only way out is a timeout. The waiting should be something like this:
if (wait_event_interruptible_timeout(dev->wait,
(!dev->rd_full),
HZ) == 0) { /* 1 second timeout */
printk(KERN_WARNING "mydriver: Timed out while flushing. Output data was lost.\n");
mutex_unlock(&dev->wr_mutex);
return -ETIMEDOUT;
}
(Relevant Linux kernel: vanilla 2.6.37)
This really is a note to future self. All of the sudden, terminal windows on my Fedora 12 ceased to respect num lock, shift and CTRL modifiers, and I somehow figured out that it may have something to do with the fact that I was having a Vmplayer machine running.
It’s noted a lot elsewhere, but I’ll write it down for myself here: To get things back to normal, just type
$ setxkbmap
at shell prompt, and all is fine again. This issue is too rare to waste the time to go to the bottom with.
So I bought a cheapo USB modem/fax on Ebay. I have to admit, that I didn’t really expect it to work with my Fedora 12 Linux machine, but it was actually detected as ACM modem, and /dev/ttyACM0 appeared. lsusb -v reveals it’s identified as ID 0572:1329 Conexant Systems (Rockwell), Inc.
Note to self: This is the chosen fax/modem, attached to the computer. As a matter of fact, I have two of these, since I tried to buy another type, but ended up with the same thing exactly.
The modem sends faxes properly and receives them properly when Bezeq dials me to deliver the fax. The command used below is OK with the cellular fax. Just change “filenameprefix” with whatever prefix is desired, and it works slowly but OK.
This tech note shows the fax session sequence, for information.
After symlinking /dev/modem to /dev/ttyACM0, I managed to get “fax send” to work properly. “fax receive” was a different story.
The logs revealed that the problematic part was
efax: 31:04 received 10 bytes: ff c8 c1 00 62 0f 01 00 ef 1c
efax: 31:04 received DCS - session format
efax: 31:04 session 196lpi 9600bps 8.5"/215mm 11"/A4 1D - - 0ms
efax: 31:04 command "+FTS=1"
efax: 31:04 waiting 3.0 s
efax: 31:04 .253 [<CR><LF>OK<CR><LF>]
efax: 31:04 response "OK"
efax: 31:04 command "+FRM=96"
efax: 31:04 waiting 6.0 s
efax: 31:10 Error: timed out after command: +FRM=96
efax: 31:10 command "+FTH=3"
efax: 31:10 waiting 3.1
Which is contrary to my previous thought that a lot of “received UNKNOWN” messages was the problem. Even if these messages have something to do with the real problem, they were still there when I got the fax to work OK.
These commands are well-documented in this document, where I found out that the command stands for receiving data in 9600 bps with no specification of training length. What happened was that something got messed up in the negotiation, leaving the phone line dry, and a timeout soon to come.
The immediate solution was to call efax directly with the -c flag set to reduce the maximal bps rate to 4800. This is painfully slow, but faxes came in OK this way:
/usr/bin/efax -r filenameprefix -c 1,1,0,2,0,0,0,0 -v ewin -v =chewmainrxtf -d/dev/modem -iZ 'i&FE0&D2S7=120' '-i&C0' -iM1L0 -l "+0 000" -kZ
This -c flag is a list of capability flags, as specified in the “man efax” manpage. The second digit, “1″, stands for maximum 4800 bps.
The weird thing is the following part in the (failed) log:
efax: 30:52 using CX93001-EIS_V0.2002-V92 in class 1
efax: 30:52 command "+FRM=?"
efax: 30:52 waiting 5.0 s
efax: 30:52 .437 [<CR><LF>3,24,48,72,73,74,96,97,98,121,122,145,146<CR><LF>]
efax: 30:52 .437 [<CR><LF>OK<CR><LF>]
efax: 30:52 response "OK"
What happens here is that the computer asks the modem for its allowed speeds, and it explicitly says it can take FRM=96 (among others). Annoying.
Anyhow, I think I’ll settle for 4800 bps for the few times I’ll ever need to receive a fax in the future. Solving this simply isn’t worth the effort.
Bonus: send command
For an extreme case with a horrible line (remote responded with EOP to an EOP from my fax, instead of an MCF, which means page OK), I opted for a 4800 bps transmission:
# /usr/bin/efax -v ewin -v chewmainrxtf -d/dev/modem -c 1,1,0,2,0,0,0,0 -x /var/lock/LCK..modem -iZ '-i&FE0&D2S7=120' '-i&C0' -iM1L0 -l '+0 000 000 0000' -kZ -t T077777777 fax-to-send.001
Replace the argument of -T with the number to dial, and the last argument with the file to send.
The header of the fax went out with a UNIX timestamp, since I didn’t state one in the command. But who cares.
While the PCI Express standard is impressive in that it actually makes sense (well, most of the time) there is a pretty annoying thing about read requests reordering.
By the way, I talk about TLP packet formation in general in another post.
In section 2.4.1 of the PCIe spec 1.1, it says that read requests may be reordered as they travel across the switching network, and same goes for read completions associated with different read requests. The read-related packets, which must arrive in the same order they were sent, are read completions associated with the same read request, or as the spec puts it: “… must return in address order”.
This would be a good time to mention, that a read request may be larger than the maximal payload size, so obviously the completer must have a means of splitting the completion into several TLPs, which must be sent in rising address order. And if we’re at it, there’s a boundary restriction on how to cut the data in pieces, namely that the cuts are on boundaries of RCB, where RCB can either be 64 bytes or 128 bytes (if you’re not a Root Complex you may cut on 128-byte boundaries only, and if you are a Root Complex, you choose 64 or 128, and configure the endpoints telling then what you chose).
And there’s a restriction on the maximal read request size, but that’s a different story.
So far the specification makes sense: Read completions will be split into several TLPs pretty often, and they have to arrive to the requester in linear address order, so these packets must not be reordered.
But what if the endpoint needs to collect a chunk of data which is larger than the maximal read request size (typically 512 bytes)? It will have to issue several read requests for that. But read requests and completions from different read requests may be reordered.
So if we want to assure that the data arrives in linear order (which is necessary when the data goes into a FIFO-like data sink) each read request can be transmitted only when the last completion TLP from the previous request arrives. Otherwise, a completion TLP from the following request may arrive before that last packet. Hence there’s a time gap of non-utilized bandwidth.
In general there is no problem having several outstanding read requests. So had read requests and read completions been strictly ordered, it would be possible to send the following read request more or less immediately after the first one, and completions would arrive continuously.
Another issue, which is less likely to bother anyone, is that if some software makes assumptions on the order at which data in some buffer is updated, this can cause rare bugs. For example, if the read completions update some dual port RAM, and there’s software reading from this buffer, it may deduce from the update of some high-addressed memory cell that the entire buffer is updated.
And a final word: Since PCIe infrastructure is pretty plain when this post is written, I will be surprised if anyone manages to catch any packet reordering taking place for real. But exactly as write reordering is commonplace in modern CPUs to increase performance, it’s only a matter of time before PCIe switching networks do the same.
Update: I got an email from someone informing me that he spotted reordering taking place on some Intel desktop chipsets. So this isn’t just theory, after all.
Questions & Comments
Since the comment section of similar posts tends to turn into a Q & A session, I’ve taken that to a separate place. So if you’d like to discuss something with me, please post questions and comments here instead. No registration is required. The comment section below is closed.
Posted Under:
PCI express
This post was written by
eli on May 24, 2011
Comments Off
