USB 3.0 device compliance test notes

Introduction

While implementing Xillybus‘ USB 3.0 general purpose IP core for FPGAs, I found the USB Implementers Forum’s compliance tool handy, yet somewhat quirky, for verifying I got things right. It was USB3CV version 2.1.12.1, running on Windows 10 @32 bit. The 64 bit version works the same (I’ve tested it as well).

A GPLed open-source version for Linux is something one could have wished for (and would probably improve things considerably), but that’s probably too much to expect when Microsoft is all over the USB standard.

These are my notes as I went along.

Obtaining and installing

Download USB3CV from this page. Installation went smooth on Windows 10 @32 bits (and 64 bits as well).

As suggested by the utility’s author, disable UAC completely on the system by invoking regedit, setting the EnableLUA to 0 on the following path: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System. And restart Windows. This seems to be a matter of convenience, and not something to do on anything else but an internal computer intended for testing only, as it’s a security hole.

A usbif.json file can be obtained from USB-IF for use by USB3CV. This only allows looking up the Vendor ID, so there’s no problem working without it. There will be two red lines in the log, but the relevant tests passes without this file as well.

Installing on Windows 7 (32 bit) failed on installing the Microsoft Visual C++ 2017 Redistributable (vc_redist.x86.exe) prerequisite. Despite removing everything installed on that computer (under Programs and Features), the problem remained — it’s probably because SP1 isn’t installed on that machine.

The hijack

When invoked, the USB3CV program replaces the original xHCI driver in Window’s I/O stack with one it uses for testing, and returns the original one when exiting. This means that all USB devices that are connected to the USB controller (possibly all USB devices on a motherboard) become effectively disconnected. That includes USB 2.0 and USB 1.1 devices. To work around this, either use good-old PS/2 keyboard and mouse, employ a remote session, or plug in an extra USB board (possibly as low as USB 1.1) and plug the USB mouse and keyboard there. Otherwise, well, no mouse and keyboard on a Windows system.

Also, be sure to close USB3CV before shutting down Windows, or it may not have enough time to bring the original driver back. See more notes on this issue below.

USB3CV is kind enough to prompt for which USB controller to take over, and it’s also fine if you accidentally knock out your own USB mouse and keyboard (a second dialog box requires confirming the takeover with a mouse click, or it times out and reverts it).

The immediate difference when plugging in a device when USB3CV is running (and has taken over the relevant controller) is that nothing happens — there is no enumeration nor descriptor fetching, as there would usually be. This happens only later on, when requesting tests, in which case the controller is scanned for devices. Several times, actually.

Another significant difference is that the test xHCI driver doesn’t have these small workaround features that a usual xHCI driver has for getting unstuck from protocol bug deadlocks, and it doesn’t attempt to smooth protocol errors. Which makes sense: A regular xHCI driver’s goal is to make the device work. The test driver is there to expose errors, so it should get stuck when things are done wrong. Hence it may expose bugs that were smoothed out when the device was connected in a regular manner to a computer. For example, a bug in the device’s LTSSM implementation may be smoothed out by a regular driver by issuing a warm reset and starting all over again, without any log message, but USB3CV’s driver will just fail.

So if weird stuff happens with USB3CV, check your own implementation, and don’t look for bugs in USB3CV. Reject the immediate “but it worked before” instinct to blame something else than your own design.

Hands on

Double-click the USB 3 Gen X CV icon. A dialog box with a list of USB controller(s) opens. Select the one that the device is attached to. All other USB devices on that controller, of all speeds, will be disconnected from the computer. Then the “Command Verifier” asks to verify this choice. If you just disabled your own mouse and keyboard you can’t click “Continue”, and that dialog box will time out, and the hijacked USB controller is released.

Then the main windows opens. Select “Chapter 9 Tests (USB 3 Gen X devices)”, Compliance Test (or Debug for individual tests), and click Run.

This is when USB3CV tries to find devices on the hijacked USB controller (it’s complete silence on wires until then). Sometimes this fails for no apparent reason — see below. A dialog box asking to select the device to work with appears, and is then followed by three dialog boxes asking for the number of retimers. If you don’t know what it’s about, you don’t have any, so select zero on all three.

When tests fail, the error messages may be misleading. For example, a problem with the device’s LTSSM made the GetDecriptor() test fail, spitting out the paragraphs in the spec that aren’t met, but on the wires there was no related SETUP packet sent (because the link wasn’t up, it turned out eventually). However the SET_SEL test went through OK nevertheless. So it may be really confusing. This is easily mistaken for a bug in USB3CV.

Even worse, it seems like a test failure can lead to all kind of unexpected and unrelated errors in following tests.

The tool also complains when the device declares itself as USB 3.0 in the device descriptor rather than USB 3.2, considering it to be a test failure. What about USB 3.0 devices, not supporting anything related to SuperSpeedPlus? Why should they even mention USB 3.2?

When things get stuck

If weird things happen (in particular if the device isn’t found by USB3CV and/or otherwise) re-run USB3CV and exit it, so the original xHCI controller is brought back upon exit. That’s what USB3CV expects to see on invocation, and it doesn’t work properly otherwise. So just run the tool and exit immediately, and then run it again.

It’s better to start USB3CV with the device already plugged in. Moving it to another plug while USB3CV is running often helps.

Crashes

Unfortunately, USB3CV crashes quite a lot (mostly in relation with test failures, in particular failing tests related to low power states). The “Command Verifier Tool has stopped working” dialog box may appear. A rather ironic workaround seems to work: Clicking “Abort” as the tests run (towards the end, actually), and then clicking “No”, a bit before the end of the tests, in the dialog box asking if you really want to abort (so the test isn’t aborted, after all). Sometimes the enumeration test is done, sometimes it isn’t (and fails with some ugly error), so maybe that’s related.

Sometimes USB3CV just gets stuck in a test, and attempting abort the test doesn’t help. Closing the USB3CV windows brings up “Wait for a stack switch” after which Windows crashes with a “Your PC ran into a problem and needs to restart. You can restart”. Which probably means that it’s OK to recycle power (no other possibility left). Windows suggest searching online for “WDF_VIOLATION” too.

Whether the power recycle took place or not, USB3CV didn’t have the opportunity to return the original xHCI driver as it usually does upon a normal exit. Therefore, be sure to invoke and exit USB3CV as mentioned above to get the system back to its original state.