To make a long story short…
There is really no need to work hard to make your FPGA talk with a PC. Xillybus gives you the end-to-end connection interfacing FPGAs with both Linux and Windows computers.
The challenge
At times, FPGA engineers are faced with the need to transfer data to or from a regular PC. Sometimes it’s the purpose of the project (e.g. data acquisition, frame grabbing, firmware loading etc.). Others need data transport with a PC for testing their HDL design on silicon. Or just run whatever automated tests or calibration processes involved in the project. This way or another, the lowest-level piece of logic (the FPGA) now needs to talk with the highest level form of logic (a user space application running in protected memory mode within a fullblown operation system). That’s quite a gap to bridge.
Since I published a short guide about the basics of PCI Express, I keep receiving questions implying that some FPGA engineers don’t grasp what’s ahead of them when they start such a project. Even for low-bandwidth assignments (where no DMA is necessary) there’s quite some way to go before having something that works in a stable manner. While Linux offers a fairly straightforward API for writing device drivers, developing any kind of driver for Windows is much of a project in itself. And even though my heart is with Linux, it’s pretty clear that “PC” and “Windows” are synonyms to most people today.
Non-PCIe solutions
There are two common approaches today for making a Windows PC and an FPGA talk:
- Using Cypress’ EZ-USB chip, which supplies a generic data interface for USB communication. Windows drivers are available from Cypress, but interfacing the chip with the FPGA requires a substantial piece of logic, as well as some 8051 firmware hacking. From my own experience and others’, those chips have some “personality” once they’re put on a real design’s board. So all in all, this is not a rose garden, and yet for many years this was considered the only sane solution.
- Connecting the FPGA to the PC’s network plug through an Ethernet chip, typically a Marvell Alaska transceiver. This solution is attractive in particular when data goes from the FPGA to the PC, since raw MAC packets are quite easy to assemble. The main problem with this solution is that even though it usually works fine, it’s just because the hardware and software components are more reliable than required, as detailed in another post of mine.
Painless PCIe
As mentioned above, the good news is that Xillybus makes it amazingly easy: On the computer, open a plain file in an user space application, using whatever programming language you want. No creepy API involved. Just read and write to a file, and put a plain FIFO inside the FPGA to hold the data.
Xillybus supports a variety if Xilinx and Altera FPGAs, regardless of the host’s operating system: All Spartan 6, Virtex-5 and Virtex-6 devices with a “T” suffix (those having a built-in PCIe hardware core). As for Altera, all devices having the hard IP form of PCI compiler for PCI Express.
The log level is set in the registry at HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Setup\LogLevel. The key may need to be created (DWORD). The default on my computer turned out to be 0x2000ff00, which means maximal logging is achieved. The set bit 29 tells the logger not to flush data into the file after each entry (faster logging, but data can be lost on crashes). Setting bit 31 will add time stamps.
On Windows 7, the file is C:\Windows\inf\setupapi.dev.log which can look like this when an unknown PCI device is detected by the system:
>>> [Device Install (Hardware initiated) - pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00\4&27574d66&0&0008]
>>> Section start 2012/02/08 20:55:17.486
ump: Creating Install Process: DrvInst.exe 20:55:17.496
ndv: Retrieving device info...
ndv: Setting device parameters...
ndv: Searching Driver Store and Device Path...
dvi: {Build Driver List} 20:55:17.506
dvi: Searching for hardware ID(s):
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
dvi: pci\ven_10ee&dev_ebeb&cc_ff0000
dvi: pci\ven_10ee&dev_ebeb&cc_ff00
dvi: Searching for compatible ID(s):
dvi: pci\ven_10ee&dev_ebeb&rev_00
dvi: pci\ven_10ee&dev_ebeb
dvi: pci\ven_10ee&cc_ff0000
dvi: pci\ven_10ee&cc_ff00
dvi: pci\ven_10ee
dvi: pci\cc_ff0000
dvi: pci\cc_ff00
cpy: Policy is set to make all digital signatures equal.
dvi: Enumerating INFs from path list 'C:\Windows\inf'
inf: Searched 0 potential matches in published INF directory
inf: Searched 35 INFs in directory: 'C:\Windows\inf'
dvi: {Build Driver List - exit(0x00000000)} 20:55:17.686
ndv: Selecting best match from Driver Store (including Device Path)...
dvi: {DIF_SELECTBESTCOMPATDRV} 20:55:17.686
dvi: No class installer for 'PCI Device'
dvi: No CoInstallers found
dvi: Default installer: Enter 20:55:17.696
dvi: {Select Best Driver}
! dvi: Selecting driver failed(0xe0000228)
dvi: {Select Best Driver - exit(0xe0000228)}
! dvi: Default installer: failed!
! dvi: Error 0xe0000228: There are no compatible drivers for this device.
dvi: {DIF_SELECTBESTCOMPATDRV - exit(0xe0000228)} 20:55:17.716
ndv: {Core Device Install} 20:55:17.716
! ndv: Installing NULL driver!
dvi: Set selected driver complete.
dvi: {DIF_ALLOW_INSTALL} 20:55:17.716
dvi: No class installer for 'PCI Device'
dvi: Default installer: Enter 20:55:17.716
dvi: Default installer: Exit
dvi: {DIF_ALLOW_INSTALL - exit(0xe000020e)} 20:55:17.716
dvi: {DIF_INSTALLDEVICE} 20:55:17.716
dvi: No class installer for 'PCI Device'
dvi: Default installer: Enter 20:55:17.716
! dvi: Installing NULL driver!
dvi: Writing common driver property settings.
dvi: {Restarting Devices} 20:55:17.736
dvi: Restart: PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008
dvi: Restart complete.
dvi: {Restarting Devices exit} 20:55:17.906
dvi: Default installer: Exit
dvi: {DIF_INSTALLDEVICE - exit(0x00000000)} 20:55:17.906
ndv: Device install status=0xe0000203
ndv: Performing device install final cleanup...
! ndv: Queueing up error report since device installation failed...
ndv: {Core Device Install - exit(0xe0000203)} 20:55:17.906
ump: Server install process exited with code 0xe0000203 20:55:17.916
<<< Section end 2012/02/08 20:55:17.916
<<< [Exit status: FAILURE(0xe0000203)]
A successful installation for the same device could look like this:
>>> [Device Install (DiShowUpdateDevice) - PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008]
>>> Section start 2012/02/08 23:22:31.407
cmd: "C:\Windows\system32\mmc.exe" C:\Windows\system32\devmgmt.msc
dvi: {DIF_UPDATEDRIVER_UI} 23:22:31.407
dvi: No class installer for 'PCI Device'
dvi: Default installer: Enter 23:22:31.423
dvi: Default installer: Exit
dvi: {DIF_UPDATEDRIVER_UI - exit(0xe000020e)} 23:22:31.423
ndv: {Update Driver Software Wizard for PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008}
inf: Opened INF: 'c:\driver\xillybus.inf' ([strings])
inf: {SetupCopyOEMInf: c:\driver\xillybus.inf} 23:22:44.652
sto: {Import Driver Package: c:\driver\xillybus.inf} 23:22:44.683
sto: Importing driver package into Driver Store:
sto: Driver Store = C:\Windows\System32\DriverStore (Online | 6.1.7600)
sto: Driver Package = c:\driver\xillybus.inf
sto: Architecture = x86
sto: Locale Name = neutral
sto: Flags = 0x00000000
sto: Copying driver package files to 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'.
inf: Opened INF: 'c:\driver\xillybus.inf' ([strings])
inf: Opened INF: 'c:\driver\xillybus.inf' ([strings])
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000000
flq: SourceRootPath - 'c:\driver\i386'
flq: SourceFilename - 'xillybus.sys'
flq: TargetDirectory- 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386'
flq: {FILE_QUEUE_COPY exit(0x00000000)}
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000000
flq: SourceRootPath - 'c:\driver'
flq: SourceFilename - 'xillybus.inf'
flq: TargetDirectory- 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'
flq: {FILE_QUEUE_COPY exit(0x00000000)}
flq: {_commit_file_queue}
flq: CommitQ DelNodes=0 RenNodes=0 CopyNodes=2
flq: {_commit_copy_subqueue}
flq: subqueue count=2
flq: source media:
flq: SourcePath - [c:\driver\i386]
flq: SourceFile - [xillybus.sys]
flq: Flags - 0x00000000
flq: {_commit_copyfile}
flq: CopyFile: 'c:\driver\i386\xillybus.sys'
flq: to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\SET89B8.tmp'
flq: MoveFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\SET89B8.tmp'
flq: to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\xillybus.sys'
flq: {_commit_copyfile exit OK}
flq: source media:
flq: SourcePath - [c:\driver]
flq: SourceFile - [xillybus.inf]
flq: Flags - 0x00000000
flq: {_commit_copyfile}
flq: CopyFile: 'c:\driver\xillybus.inf'
flq: to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\SET89D8.tmp'
flq: MoveFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\SET89D8.tmp'
flq: to: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf'
flq: {_commit_copyfile exit OK}
flq: {_commit_copy_subqueue exit OK}
flq: {_commit_file_queue exit OK}
pol: {Driver package policy check} 23:22:44.870
pol: {Driver package policy check - exit(0x00000000)} 23:22:44.870
sto: {Stage Driver Package: C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf} 23:22:44.870
inf: Opened INF: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf' ([strings])
inf: Opened INF: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf' ([strings])
sto: Copying driver package files:
sto: Source Path = C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}
sto: Destination Path = C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000010
flq: SourceRootPath - 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386'
flq: SourceFilename - 'xillybus.sys'
flq: TargetDirectory- 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386'
flq: {FILE_QUEUE_COPY exit(0x00000000)}
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000010
flq: SourceRootPath - 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}'
flq: SourceFilename - 'xillybus.inf'
flq: TargetDirectory- 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}'
flq: {FILE_QUEUE_COPY exit(0x00000000)}
flq: {_commit_file_queue}
flq: CommitQ DelNodes=0 RenNodes=0 CopyNodes=2
flq: {_commit_copy_subqueue}
flq: subqueue count=2
flq: source media:
flq: SourcePath - [C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386]
flq: SourceFile - [xillybus.sys]
flq: Flags - 0x00000000
flq: {_commit_copyfile}
flq: CopyFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\i386\xillybus.sys'
flq: to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\SET8A54.tmp'
flq: MoveFile: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\SET8A54.tmp'
flq: to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386\xillybus.sys'
flq: {_commit_copyfile exit OK}
flq: source media:
flq: SourcePath - [C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}]
flq: SourceFile - [xillybus.inf]
flq: Flags - 0x00000000
flq: {_commit_copyfile}
flq: CopyFile: 'C:\Users\tester\AppData\Local\Temp\{35bcaf49-eb0d-78a2-1a0a-42414908c360}\xillybus.inf'
flq: to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\SET8A55.tmp'
flq: MoveFile: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\SET8A55.tmp'
flq: to: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\xillybus.inf'
flq: {_commit_copyfile exit OK}
flq: {_commit_copy_subqueue exit OK}
flq: {_commit_file_queue exit OK}
sto: {DRIVERSTORE_IMPORT_NOTIFY_VALIDATE} 23:22:44.901
! sto: Driver package does not contain a catalog file, but user wants to install anyway.
sto: {DRIVERSTORE_IMPORT_NOTIFY_VALIDATE exit(0x00000000)} 23:22:46.196
sto: Verified driver package signature:
sto: Digital Signer Score = 0xFF000000
sto: Digital Signer Name = <unknown>
sto: {DRIVERSTORE_IMPORT_NOTIFY_BEGIN} 23:22:46.196
inf: Opened INF: 'C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\xillybus.inf' ([strings])
sto: Create system restore point:
sto: Description = Device Driver Package Install: Xillybus Ltd
sto: Time = 7394ms
sto: Status = 0x00000000 (SUCCESS)
sto: {DRIVERSTORE_IMPORT_NOTIFY_BEGIN: exit(0x00000000)} 23:22:53.606
sto: Importing driver package files:
sto: Source Path = C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}
sto: Destination Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176
sto: {Copy Directory: C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}} 23:22:53.606
sto: Target Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176
sto: {Copy Directory: C:\Windows\System32\DriverStore\Temp\{296beaae-939f-1376-4169-f771e772a021}\i386} 23:22:53.622
sto: Target Path = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386
sto: {Copy Directory: exit(0x00000000)} 23:22:53.622
sto: {Copy Directory: exit(0x00000000)} 23:22:53.622
sto: {Index Driver Package: C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf} 23:22:53.622
idb: Registered driver store entry 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
idb: Published 'xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' to 'C:\Windows\INF\oem2.inf'
idb: Published driver store entry 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
sto: Published driver package INF 'oem2.inf' was changed.
sto: Active published driver package is 'xillybus.inf_x86_neutral_c6abbde6e922f176'.
sto: {Index Driver Package: exit(0x00000000)} 23:22:54.495
sto: {DRIVERSTORE_IMPORT_NOTIFY_END} 23:22:54.495
sto: Commit system restore point:
sto: Description = Device Driver Package Install: Xillybus Ltd
sto: Time = 16ms
sto: Status = 0x00000000 (SUCCESS)
sto: {DRIVERSTORE_IMPORT_NOTIFY_END: exit(0x00000000)} 23:22:54.511
sto: {Stage Driver Package: exit(0x00000000)} 23:22:54.511
ndv: Doing device matching lookup!
inf: Opened INF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
inf: Saved PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.PNF' (Language = 0409)
ndv: Found device that matches new INF!
sto: Driver package was staged to Driver Store. Time = 9766 ms
sto: Imported driver package into Driver Store:
sto: Filename = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
sto: Time = 9875 ms
sto: {Import Driver Package: exit(0x00000000)} 23:22:54.558
inf: Opened INF: 'c:\driver\xillybus.inf' ([strings])
inf: Driver Store location: C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
inf: Published Inf Path: C:\Windows\INF\oem2.inf
inf: Opened INF: 'c:\driver\xillybus.inf' ([strings])
inf: OEM source media location: c:\driver\
inf: {SetupCopyOEMInf exit (0x00000000)} 23:22:54.729
dvi: Searching for hardware ID(s):
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
dvi: pci\ven_10ee&dev_ebeb&cc_ff0000
dvi: pci\ven_10ee&dev_ebeb&cc_ff00
dvi: Searching for compatible ID(s):
dvi: pci\ven_10ee&dev_ebeb&rev_00
dvi: pci\ven_10ee&dev_ebeb
dvi: pci\ven_10ee&cc_ff0000
dvi: pci\ven_10ee&cc_ff00
dvi: pci\ven_10ee
dvi: pci\cc_ff0000
dvi: pci\cc_ff00
inf: Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
sig: {_VERIFY_FILE_SIGNATURE} 23:22:54.745
sig: Key = xillybus.inf
sig: FilePath = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
! sig: No installed catalogs matching catalog name '' were found that validated the file.
! sig: Error 1168: Element not found.
sig: {_VERIFY_FILE_SIGNATURE exit(0x00000490)} 23:22:54.761
dvi: Selected driver installs from section [Xillybus_Inst] in 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'.
dvi: Class GUID of device changed to: {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07}.
dvi: Set selected driver complete.
dvi: {Plug and Play Service: Device Install for PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008}
ump: Creating Install Process: DrvInst.exe 23:22:54.761
ndv: Infpath=C:\Windows\INF\oem2.inf
ndv: DriverNodeName=xillybus.inf:MSFT.NTx86:Xillybus_Inst:1.0.0.0:pci\ven_10ee&dev_ebeb
ndv: DriverStorepath=C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
ndv: Building driver list from driver node strong name...
dvi: Searching for hardware ID(s):
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee&rev_00
dvi: pci\ven_10ee&dev_ebeb&subsys_ebeb10ee
dvi: pci\ven_10ee&dev_ebeb&cc_ff0000
dvi: pci\ven_10ee&dev_ebeb&cc_ff00
dvi: Searching for compatible ID(s):
dvi: pci\ven_10ee&dev_ebeb&rev_00
dvi: pci\ven_10ee&dev_ebeb
dvi: pci\ven_10ee&cc_ff0000
dvi: pci\ven_10ee&cc_ff00
dvi: pci\ven_10ee
dvi: pci\cc_ff0000
dvi: pci\cc_ff00
inf: Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
sig: {_VERIFY_FILE_SIGNATURE} 23:22:54.807
sig: Key = xillybus.inf
sig: FilePath = C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf
! sig: No installed catalogs matching catalog name '' were found that validated the file.
! sig: Error 1168: Element not found.
sig: {_VERIFY_FILE_SIGNATURE exit(0x00000490)} 23:22:54.823
dvi: Selected driver installs from section [Xillybus_Inst] in 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'.
dvi: Class GUID of device changed to: {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07}.
dvi: Set selected driver complete.
ndv: {Core Device Install} 23:22:54.823
inf: Opened INF: 'C:\Windows\INF\oem2.inf' ([strings])
inf: Saved PNF: 'C:\Windows\INF\oem2.PNF' (Language = 0409)
ndv: Class {eb32c6d5-2d4d-4d8c-a83b-4db9621fdb07} does not exist.
dvi: {Installing Class}
inf: Opened PNF: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
dvi: Installing device class: 'Xillybus' {EB32C6D5-2D4D-4d8c-A83B-4DB9621FDB07}.
dvi: {Install Class Inf Section [ClassInstall32]}
dvi: {Install Class Inf Section [ClassInstall32] exit (0x00000000)}
dvi: {_SCAN_FILE_QUEUE}
flq: ScanQ flags=22
flq: SPQ_SCAN_FILE_VALIDITY
flq: SPQ_SCAN_PRUNE_COPY_QUEUE
flq: ScanQ number of copy nodes=0
flq: ScanQ action=2 DoPruning=32
flq: ScanQ end Validity flags=22 CopyNodes=0
dvi: {_SCAN_FILE_QUEUE exit(0, 0x00000000)}
dvi: {Install Class Inf Section [ClassInstall32]}
inf: Addreg=XillybusClassReg (xillybus.inf line 14)
dvi: {Install Class Inf Section [ClassInstall32] exit (0x00000000)}
dvi: {Install Class Inf Section [ClassInstall32.Services]}
dvi: {Install Class Inf Section [ClassInstall32.Services] exit (0x00000000)}
dvi: Class install completed with no errors.
dvi: {Installing Class exit(0x00000000)}
dvi: {DIF_ALLOW_INSTALL} 23:22:54.854
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: No CoInstallers found
dvi: Default installer: Enter 23:22:54.854
dvi: Default installer: Exit
dvi: {DIF_ALLOW_INSTALL - exit(0xe000020e)} 23:22:54.854
ndv: Installing files...
dvi: {DIF_INSTALLDEVICEFILES} 23:22:54.854
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: Default installer: Enter 23:22:54.854
dvi: {Install FILES}
inf: Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
inf: {Install Inf Section [Xillybus_Inst.NT]}
inf: CopyFiles=Xillybus.CopyFiles (xillybus.inf line 53)
cpy: Open PnpLockdownPolicy: Err=2. This is OK. Use LockDownPolicyDefault
flq: QueueSingleCopy...
flq: Inf : 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'
flq: SourceInf: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf'
flq: SourceSection: [sourcedisksfiles]
flq: Source root path based on SourceInf
flq: SourceRootPath: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176'
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000000
flq: {FILE_QUEUE_COPY}
flq: CopyStyle - 0x00000000
flq: SourceRootPath - 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176'
flq: SourcePath - '\i386'
flq: SourceFilename - 'xillybus.sys'
flq: TargetDirectory- 'C:\Windows\system32\DRIVERS'
flq: TargetFilename - 'xillybus.sys'
flq: SourceDesc - 'Xillybus install disk'
flq: {FILE_QUEUE_COPY exit(0x00000000)}
flq: {FILE_QUEUE_COPY exit(0x00000000)}
inf: {Install Inf Section [Xillybus_Inst.NT] exit (0x00000000)}
dvi: Processing co-installer registration section [Xillybus_Inst.NT.CoInstallers].
inf: {Install Inf Section [Xillybus_Inst.NT.CoInstallers]}
inf: {Install Inf Section [Xillybus_Inst.NT.CoInstallers] exit (0x00000000)}
dvi: Co-installers registered.
dvi: {Install INTERFACES}
dvi: Installing section [Xillybus_Inst.NT.Interfaces]
dvi: {Install INTERFACES exit 00000000}
dvi: {Install FILES exit (0x00000000)}
dvi: Default installer: Exit
dvi: {DIF_INSTALLDEVICEFILES - exit(0x00000000)} 23:22:54.885
ndv: Pruning file queue...
dvi: {_SCAN_FILE_QUEUE}
flq: ScanQ flags=620
flq: SPQ_SCAN_PRUNE_COPY_QUEUE
flq: SPQ_SCAN_FILE_COMPARISON
flq: SPQ_SCAN_ACTIVATE_DRP
flq: ScanQ number of copy nodes=1
flq: ScanQ action=200 DoPruning=32
flq: ScanQ end Validity flags=620 CopyNodes=1
dvi: {_SCAN_FILE_QUEUE exit(0, 0x00000000)}
ndv: Committing file queue...
flq: {_commit_file_queue}
flq: CommitQ DelNodes=0 RenNodes=0 CopyNodes=1
flq: {SPFILENOTIFY_STARTQUEUE}
flq: {SPFILENOTIFY_STARTQUEUE - exit(0x00000001)}
flq: {_commit_copy_subqueue}
flq: subqueue count=1
flq: {SPFILENOTIFY_STARTSUBQUEUE}
flq: {SPFILENOTIFY_STARTSUBQUEUE - exit(0x00000001)}
flq: source media:
flq: Description - [Xillybus install disk]
flq: SourcePath - [C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386]
flq: SourceFile - [xillybus.sys]
flq: Flags - 0x00000000
flq: {SPFQNOTIFY_NEEDMEDIA}
flq: {SPFILENOTIFY_NEEDMEDIA}
flq: {SPFILENOTIFY_NEEDMEDIA - exit(0x00000001)}
flq: {SPFQNOTIFY_NEEDMEDIA - returned 0x00000001}
flq: source media: SPFQOPERATION_DOIT
flq: {_commit_copyfile}
flq: {SPFILENOTIFY_STARTCOPY}
ndv: Saving LastKnownGood file C:\Windows\system32\DRIVERS\xillybus.sys (copy)
flq: {SPFILENOTIFY_STARTCOPY - exit(0x00000001)}
flq: CopyFile: 'C:\Windows\System32\DriverStore\FileRepository\xillybus.inf_x86_neutral_c6abbde6e922f176\i386\xillybus.sys'
flq: to: 'C:\Windows\system32\DRIVERS\SETB164.tmp'
cpy: CopyFile Drp is active
cpy: Source File 'C:\Windows\system32\DRIVERS\SETB164.tmp' is NOT signed NT5 Crypto.
cpy: DrpGetFileProt Status=2 dwClass=0
flq: MoveFile: 'C:\Windows\system32\DRIVERS\SETB164.tmp'
flq: to: 'C:\Windows\system32\DRIVERS\xillybus.sys'
cpy: DrpSetRegFileProt 'C:\Windows\system32\DRIVERS\xillybus.sys' Status=0 Legacy
flq: Caller applied security to file 'C:\Windows\system32\DRIVERS\xillybus.sys'.
flq: {SPFILENOTIFY_ENDCOPY}
flq: {SPFILENOTIFY_ENDCOPY - exit(0x00000001)}
flq: {_commit_copyfile exit OK}
flq: {SPFILENOTIFY_ENDSUBQUEUE}
flq: {SPFILENOTIFY_ENDSUBQUEUE - exit(0x00000001)}
flq: {_commit_copy_subqueue exit OK}
flq: {SPFILENOTIFY_ENDQUEUE}
flq: {SPFILENOTIFY_ENDQUEUE - exit(0x00000001)}
flq: {_commit_file_queue exit OK}
ndv: Registering CoInstallers...
dvi: {DIF_REGISTER_COINSTALLERS} 23:22:54.979
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: Default installer: Enter 23:22:54.995
inf: Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
inf: {Install Inf Section [Xillybus_Inst.NT.CoInstallers]}
inf: {Install Inf Section [Xillybus_Inst.NT.CoInstallers] exit (0x00000000)}
dvi: Co-installers registered.
dvi: Default installer: Exit
dvi: {DIF_REGISTER_COINSTALLERS - exit(0x00000000)} 23:22:54.995
ndv: Installing interfaces...
dvi: {DIF_INSTALLINTERFACES} 23:22:54.995
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: No CoInstallers found
dvi: Default installer: Enter 23:22:54.995
dvi: {Install INTERFACES}
inf: Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
dvi: Installing section [Xillybus_Inst.NT.Interfaces]
dvi: {Install INTERFACES exit 00000000}
dvi: Default installer: Exit
dvi: {DIF_INSTALLINTERFACES - exit(0x00000000)} 23:22:54.995
ndv: Installing device...
dvi: {DIF_INSTALLDEVICE} 23:22:54.995
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: Default installer: Enter 23:22:54.995
dvi: {Install DEVICE}
inf: Opened PNF: 'c:\windows\system32\driverstore\filerepository\xillybus.inf_x86_neutral_c6abbde6e922f176\xillybus.inf' ([strings])
dvi: Processing Registry/Property directives...
inf: {Install Inf Section [Xillybus_Inst.NT]}
inf: {Install Inf Section [Xillybus_Inst.NT] exit (0x00000000)}
inf: {Install Inf Section [Xillybus_Inst.NT.Hw]}
inf: Empty section
inf: {Install Inf Section [Xillybus_Inst.NT.Hw] exit (0x00000000)}
dvi: {Writing Device Properties}
dvi: Provider name=Xillybus Ltd
dvi: DriverDate 02/01/2012
dvi: DriverVersion=1.0.0.0
dvi: Class name=Xillybus
dvi: Manufacturer=Xillybus Ltd
dvi: Matching DeviceID=pci\ven_10ee&dev_ebeb
dvi: Strong Name=oem2.inf:MSFT.NTx86:Xillybus_Inst:1.0.0.0:pci\ven_10ee&dev_ebeb
dvi: {Writing Device Properties - Complete}
inf: {Install Inf Section [Xillybus_Inst.NT.Services]}
inf: AddService=Xillybus,0x00000002,Xillybus_Service,Xillybus_LogInst (xillybus.inf line 73)
inf: ServiceType=1 (xillybus.inf line 77)
inf: StartType=3 (xillybus.inf line 78)
inf: ErrorControl=1 (xillybus.inf line 79)
inf: ServiceBinary=C:\Windows\system32\DRIVERS\xillybus.sys (xillybus.inf line 80)
inf: DisplayName="Xillybus driver service for generic FPGA interface" (xillybus.inf line 76)
dvi: Add Service: Created service 'Xillybus'.
inf: AddReg=Xillybus_Log_Addreg (xillybus.inf line 83)
inf: {Install Inf Section [Xillybus_Inst.NT.Services] exit(0x00000000)}
dvi: Updated reflected section names for: oem2.inf
dvi: {Install DEVICE exit (0x00000000)}
dvi: Writing common driver property settings.
dvi: DriverDescription=Xillybus driver for generic FPGA interface
dvi: DeviceDisplayName=Xillybus driver for generic FPGA interface
dvi: {Restarting Devices} 23:22:57.319
dvi: Restart: PCI\VEN_10EE&DEV_EBEB&SUBSYS_EBEB10EE&REV_00\4&27574D66&0&0008
dvi: Restart complete.
dvi: {Restarting Devices exit} 23:23:08.208
dvi: Default installer: Exit
dvi: {DIF_INSTALLDEVICE - exit(0x00000000)} 23:23:08.208
dvi: {DIF_NEWDEVICEWIZARD_FINISHINSTALL} 23:23:08.208
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: Default installer: Enter 23:23:08.208
dvi: Default installer: Exit
dvi: {DIF_NEWDEVICEWIZARD_FINISHINSTALL - exit(0xe000020e)} 23:23:08.208
ndv: Device install status=0x00000000
ndv: Performing device install final cleanup...
ndv: {Core Device Install - exit(0x00000000)} 23:23:08.208
ump: Server install process exited with code 0x00000000 23:23:08.224
ump: {Plug and Play Service: Device Install exit(00000000)}
dvi: {DIF_NEWDEVICEWIZARD_FINISHINSTALL} 23:23:08.239
dvi: No class installer for 'Xillybus driver for generic FPGA interface'
dvi: No CoInstallers found
dvi: Default installer: Enter 23:23:08.239
dvi: Default installer: Exit
dvi: {DIF_NEWDEVICEWIZARD_FINISHINSTALL - exit(0xe000020e)} 23:23:08.239
ndv: {Update Driver Software Wizard exit(00000000)}
<<< Section end 2012/02/08 23:23:11.281
<<< [Exit status: SUCCESS
The helpful information here is mostly which parts of the INF file were read and interpreted, and what was actually installed (a new class in this case).
If you’re reading this, it’s likely that you’ve experienced a bug check (blue screen of death, BSOD, if you like) as a result of calling IoConnectInterruptEx().
It’s also likely that you used this function to support MSI (Message Signaling Interrupt). You may have attempted to follow the horribly misleading example given by Microsoft itself.
The thing is, that there’s a parameter they omitted in that example, and that’s params.MessageBased.ConnectionContext.InterruptMessageTable. That’s a pointer to a PIO_INTERRUPT_MESSAGE_INFO. If you don’t know what I’m talking about, I won’t bother to explain that: It already means I’ve just solved your problem.
I’ll just say, that IoConnectInterruptEx() uses that parameter as a pointer to write the value of another pointer of some all-so-cute and most likely useless structure of information. So if this parameter isn’t initialized, the kernel attempts to write to a null pointer, and there we are with a blue screen.
To get the example wrong in an official page is somewhat understandable. Using a pointer without checking it for being null in a kernel function is completely amazing.
While porting Xillybus to Virtex-5, I ran into nasty trouble. In the beginning, it looked like the MSI interrupt delivery mechanism was wrong, and then it turned out that the core gets locked up completely after a few packets, and refuses to send any TLPs after a few sent. I also noticed that the PCIe core has the “Fatal Error Detected” flag set in its status register (or more precisely, Xillybus banged me in the head with the bad news). Eventually, I found myself resetting the core with a debounced pushbutton connected to sys_reset_n at some very certain point in the host’s boot process to make the system work. Using just PERST_B, like the user guide suggests, simply didn’t work.
All this was with version 1.15 of the PCIe endpoint block plus, which was introduced in ISE 13.2. Quite by chance, I tried ISE 13.1, which comes with version 1.14 of the core. And guess what, suddenly PERST_B connected to sys_reset_n did the job, and the Fatal Error vanished.
I have to admit I’m quite amazed by this.
Questions & Comments
Since the comment section of similar posts tends to turn into a Q & A session, I’ve taken that to a separate place. So if you’d like to discuss something with me, please post questions and comments here instead. No registration is required. The comment section below is closed.
Posted Under:
FPGA,
PCI express
This post was written by
eli on December 20, 2011
Comments Off
In Verilog there’s a bit shifter operator, which isn’t used a lot, since FPGA designers prefer to state exact bit vectors. But sometimes bit shifting makes the code significantly more readable. Too bad that Xilinx’ XST synthesizer doesn’t get it right in a specific case.
Namely, the following statement is perfectly legal:
always @(posedge clk)
reduce <= 1 + (end_offset >> (6 + rcb_is_128_bytes - format_shift) );
But it turns out that Xilinx ISE 13.2 XST synthesizer gets confused by the calculation of the shift rate, and creates something wrong. I can’t even tell what it did, but it was wrong.
So the rule is simple: It’s fine to have the shift number being a register (even combinatoric) or a wire, but no inline calculations. So this is fine:
always @(format_shift or rcb_is_128_bytes)
if (rcb_is_128_bytes)
case (format_shift)
0: shifter <= 7;
1: shifter <= 6;
default: shifter <= 5;
endcase
else
case (format_shift)
0: shifter <= 6;
1: shifter <= 5;
default: shifter <= 4;
endcase
always @(posedge clk)
reduce <= 1 + (end_offset >> shifter );
(assuming that format_shift goes from zero to 2).
Actually, I would bet that it’s equally fine to calculate the number of shifts and put the result in a wire. I went for the case statement hoping that the synthesizer will take the hint that not all values that fit into the registers are possible, and will hence avoid implementing impossible shift values.
Needless to say, I know about this because something went horribly wrong all of the sudden. I believe XST version 12.2 handled the shift calculation OK. And then people ask me why I don’t like upgrades.
reduce_header_credits <= 1 + (effective_end_offset >> (6 + rcb_is_128_bytes – recvbuf_format_shift) )
There are several ways to stop these pingbacks, 95% of which are spam.
My method may not be optimal, but has the elegance of simplicity. Simply edit the part in the end of wp-trackback.php (at WordPress’ root directory) going
if ( !empty($tb_url) && !empty($title) ) {
header('Content-Type: text/xml; charset=' . get_option('blog_charset') );
if ( !pings_open($tb_id) )
trackback_response(1, 'Sorry, trackbacks are closed for this item.');
$title = wp_html_excerpt( $title, 250 ).'...';
$excerpt = wp_html_excerpt( $excerpt, 252 ).'...';
[ ... snipped ... ]
trackback_response(0);
}
and turn it into
trackback_response(1, 'Sorry, trackbacks are closed for this item.');
(don’t remove the trailing “?>”)
So this gives the correct response to the client trying to send trackbacks, and doesn’t bother you anymore.
Just lucky?
I’ve been approached a few times with requests to design the FPGA part of an FPGA-to-PC link over Ethernet. The purpose of the link is typically transporting a large amount of data to the PC. The application varies from continuous data acquisition to frame grabbing or transport of a raw video image stream. What these applications have in common, is that the client expects a reliable, easy-to-implement data channel. Just send the packets to the broadcast MAC address, and you’re done.
When doubting the reliability of this solution, I usually get the “I know from previous projects that it works” argument. I can’t argue with their previous success. But there’s a subtle difference between “it works” and “it’s guaranteed to work”. To a serious FPGA engineer, that’s the difference between “maybe I was lucky this time” and “this is a project I’m ready to release”.
Ethernet is inherently unreliable
The most important thing to know about Ethernet (in any data rate) is that is was never meant to be reliable. As a physical layer for networks, the underlying assumption is that a protocol layer detects packet drops and issues retransmissions as necessary. Put simply, this means that an Ethernet chip that drops a packet every now and then is considered 100% OK.
Since packet losses cause a certain data rate performance hit (e.g. TCP/IP streams will be halted for a short period), efforts are made to keep them to a minimum. For example, the Ethernet 802.3 standard states 10-10 as the objective for the raw bit error rate on a copper wire Gigabit Ethernet link (1000BASE-T, see section 40.1.1). That means that a packet drop every 10 seconds is considered within the standard’s objectives. Packet drops may also occur on the operating system level: The network stack may take the freedom to drop packets just because they didn’t arrive at a good time. This happens less when the computer is generally idle (i.e. in the lab) but may become more evident under load (that is, in real-life use).
Prototype vs. production
Engineers are often mislead to think that the link is reliable because they can’t see any packet drops on the particular prototype they’re working on. It’s also easy to overlook sporadic packet drops during the development stages. The problem becomes serious when reaching the production stage, when a no-errors system needs to be put on the table. Even worse, production copies of the system may suddenly start to fail once in a few hours or so. The QA tests may not spot these issues, so the complaints may come from end-users feeling there’s something wrong with their devices, which the vendor has no clue about. I mean, imagine your car’s dashboard going crazy for a second once a month, and the vendor insisting on that being impossible. Would you stay with that car?
Working it around
The natural way to work around Ethernet packet drops is either accepting the data loss or implementing a retransmission mechanism.
Living with the data loss is possible in e.g. one-shot data acquisition applications, when the trigger is recurrent. Say, if a single frame is grabbed from a video stream, and it’s OK to fail on the first attempt, that’s fine. As long as nobody feels the unexpected delay of 1/30th of a second.
Retransmissions may be significantly trickier, in particular if the data goes from the FPGA to the PC. The thing is, that it will take some time for the PC to respond on the lost packet, and that time may be unlimited. For example, in today’s Linux implementations, the analysis of network packets is done in a tasklet context, and not by the interrupt service routine. Since tasklets are merely scheduled as a high-priority process, the latency until the packets are analyzed closely enough to detect a packet loss depends on how busy the computer is at that time.
One could hack the Ethernet card’s device driver to check a special field in each packet (say, a counter). Let’s say that the packet interrupt is handled within 10 μs, and that the packet loss is reported back to the FPGA in no time. This means it has to store 10 kbits worth of previous packets (at least) to support a Gigabit link. Actually, that’s fine. A Xilinx FPGA’s internal RAM is more or less of that size. Too bad it’s not realistic.
And that’s because the underlying assumption of 10 μs response time is problematic, since any other kernel component can turn off interrupts while minding its own business (typically holding a spinlock). This other component could be completely unrelated to the Ethernet application (a sound card driver?) and not be active at all when the link with the FPGA is tested in QA. And it could be something not happening very often, so the sudden high latency becomes a rare bug to handle.
So taking a more realistic approach, it’s more like storing several megabytes of data to make sure all packets stay in memory until their safe arrival has been confirmed. This involves a controller for an external memory (a DDR SDRAM, I suppose) and some nontrivial state machine for keeping track of the packet traffic. While meeting the requirement of being reliable, it’s not really easy to implement. Well, easy to implement on a computer, which is the natural user of an Ethernet link. Not an FPGA.
The right way to do it
The correct thing to do, is to use a link which was originally intended for communication between a computer and its peripheral. Several interfaces exist, but today the most appealing one is PCI Express, as it’s expected to be supported for many years ahead. Being the successor of good old PCI, its packet relay interface guarantees true reliability, which is assured by the PCIe fabric’s hardware.
The PCIe solution is often avoided because of the complexity of setting up the FPGA logic for transmission over the bus. This is no excuse in situations where Xillybus fits the scenario, as it provides a simple interface on both sides for transmission of data between an FPGA and its Linux host. If that’s not the case, the two choices are either to suck it up and write the PCIe interface yourself, or revert to using Ethernet, hoping for the best.
Summary
I always say that it’s fairly easy to get an FPGA doing what you want on a lab prototype. There are a lot of engineers not asking for too much money out there, who will do that for you. But when it comes to designing the system so it’s guaranteed to work, that’s a whole different story. That includes using correct design techniques in the FPGA logic’s HDL, constraining the timing correctly, ensuring that the timing requirements of the surrounding hardware are met, as defined in their datasheets etc. The difference isn’t seen at the prototype level, so quick and dirty FPGA work gets away with it in the beginning. It’s during the later stages, when temperature checks are run and the electronics is being duplicated, that things start to happen.
At that point people tend to blame the FPGA for being an unreliable solution. Others adopt mystic rules such as the 9o%-10% rule, basically saying that the real work starts in the last phase.
But the truth is that that if you got it right in the first place, there’s no reason why things should go wrong during production. If the prototype was done professionally, turning it into a product is really not a big deal. And having said that, yes, sometimes people do everything wrong, and just turn out lucky.
As for using Ethernet as a reliable link, it all boils down to if you want to gamble on it.
The goal
Use my Microsoft mouse’s two extra buttons to do Copy and Paste on a Fedora 12 machine (Gnome 2.28.2). The problem is that different applications have different keystrokes. But the nice thing is that almost all applications will respond properly to Alt-e c for copy (that is, open the “Edit” menu, and choose C”). CTRL-C is problematic, because pressing it accidentally over a shell will halt the execution of whatever is running.
I should note that this doesn’t work with Xemacs. I’ll come to that some day, I suppose.
Doing it
Assuming the necessary packages are installed (see below), run xev and press the relevant mouse buttons over the window to detect which button generates what event.
Then setup xbindkeys’ initial configuration file (and wipe any existing settings, if present) with
xbindkeys --defaults > /home/eli/.xbindkeysrc
and edit the file, commenting out all examples, just to be safe.
Then add the following snippet to the same file:
# Copy with mouse
"xte 'keydown Alt_L' 'key e' 'keyup Alt_L' 'usleep 10000' 'key c'"
release + b:8
# Paste with mouse
"xte 'keydown Alt_L' 'key e' 'keyup Alt_L' 'usleep 10000' 'key p'"
release + b:9
Note that the relevant mouse buttons were found to be 8 and 9, using xev. The “release” keyword makes sure that the script is run upon release events, and not when the button is still pressed, because the target applications won’t execute the command while the button is pressed. The 10ms sleep between opening the menu and the command was necessary for Firefox to catch the command. Tweaking is the name of the game.
And finally, if xbindkeys isn’t run, just go
$ killall xbindkeys ; xbindkeys
Note that there’s no need to rerun xbindkeys after modifying its configuration file, as it’s reloaded automatically.
To have xbindkeys executed on each login (on a Gnome system) add the two following lines to /etc/gdm/PreSession/Default:
su -l $USER -c killall xbindkeys
su -l $USER -c xbindkeys
Just a list of relevant utilities
(Not all were used for the original purpose)
- xev — Creates a small window, and prints out events related to it. Useful for mapping the codes of mouse events, for example
- xbindkeys — A daemon which catches key or mouse events, and executes shell commands accordingly (never tried it, though. yum install xbindkeys)
- xmodmap — Utility for modifying the mapping of keys, e.g. reverse the mouse buttons or certain keys on the keyboard (but not their function)
- xte and xautomation — Create fake key presses for running X applications automatically (yum install xautomation)
In short: I’ve known for a long time that OpenGL wasn’t working on my graphics card, and the display was indeed sluggish.
The problem is most easily shown by going
$ glxinfo | grep OpenGL
OpenGL vendor string: Mesa Project
OpenGL renderer string: Software Rasterizer
OpenGL version string: 2.1 Mesa 7.7.1-DEVEL
OpenGL shading language version string: 1.20
OpenGL extensions:
as shown in http://www.x.org/wiki/radeonBuildHowTo it gives.
A much more interesting output came from
# LIBGL_DEBUG=verbose glxinfo | grep openGL
libGL: OpenDriver: trying /usr/lib64/dri/r600_dri.so
libGL error: dlopen /usr/lib64/dri/r600_dri.so failed (/usr/lib64/dri/r600_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: r600_dri.so
libGL error: driver pointer missing
libGL: OpenDriver: trying /usr/lib64/dri/swrast_dri.so
Aha! Looks like yet another punishment for running 64 bit! And indeed, the relevant file isn’t to be found on my system, so the software rasterizer is loaded instead.
According to this forum answer, I need to upgrade libdrm_radeon.
$ yum provides '*/r600_dri.so'
gave the answer: It’s in mesa-dri-drivers-experimental-7.6-0.13.fc12.x86_64. Do I want to use something experimental? Hmmm…
But checking up Mesa’s site, it looks like what they consider experimental is what is usually considered production. Judging from the bugs they fix afterwards, that is.
# yum install mesa-dri-drivers-experimental
Logged out and in again (restart X), and tried:
$ glxinfo | grep OpenGL
OpenGL vendor string: Advanced Micro Devices, Inc.
OpenGL renderer string: Mesa DRI R600 (RV710 954F) 20090101 TCL DRI2
OpenGL version string: 1.5 Mesa 7.7.1-DEVEL
OpenGL extensions
Yay! Went to System > Preferences > Desktop effects and enabled 3D acceleration. And some silly effects, to see it’s actually working.
Following http://phoronix.com/forums/showthread.php?20186-Software-Rasterizer-with-and-without-KMS I added “eli” to the “video” group. Not clear if this was necessary.
Update (June, 2014): When upgrading to kernel 3.12, Google Chrome complained about the GPU thread being stuck, and timed out after 10 seconds (e.g. on Facebook’s main page). The solution was to upgrade libdrm to 2.4.54 by compiling from sources, using
./configure --prefix=/usr --libdir=/usr/lib64/
This was probably needed because of an update in the kernel’s drm module.
Upgrading Mesa to 10.1.4 turned out to be a pain in the bottom because of lots of dependencies that needed to be downloaded. All in all, it had to be reverted by reinstalling the packages from the yum repo. It improved nothing, but windows didn’t redraw properly (for example, after issuing a command on gnome-terminal, nothing was updated until the window was moved with the mouse).
These are a couple of examples of SDMA assembly code, which performs data copy using the DMA functional unit. The first one shows how to copy data from application memory space to SDMA memory. The second example copies data from one application memory chunk to another, and hence works as an offload memcpy().
To actually use this code and generally understand what’s going on here, I’d warmly suggest reading a previous post of mine about SDMA assembly code, which also explains how to compile the code and gives the context for the C functions given below.
Gotchas
- Never let either the source address nor the destination address cross a 32-byte boundary during a burst from or to the internal FIFO. Even though I haven’t seen this restriction in the official documentation, several unexplained misbehaviors have surfaces when allowing this happen, in particular when accessing EIM. So just don’t.
- When accessing EIM, the EIM’s maximal burst length must be set to allow 32 bytes in one burst with the BL parameter, or data gets corrupted.
Application space memory to SDMA space
The assembly code goes
$ ./sdma_asm.pl app2sdma.asm
| # Always in context (not altered by script):
| #
| # r4 : Physical address to source in AP memory space
| # r6 : Address in SDMA space to copy to
| # r7 : Number of DWs to copy
| #
| # Both r4 and r5 must be DW aligned.
| # Note that prefetching is allowed, so up to 8 useless DWs may be read.
|
| # First, load the status registers into SDMA space
| start:
0000 6c20 (0110110000100000) | stf r4, 0x20 # To MSA, prefetch on, address is nonfrozen
0001 008f (0000000010001111) | mov r0, r7
0002 018e (0000000110001110) | mov r1, r6
0003 7803 (0111100000000011) | loop postloop, 0
0004 622b (0110001000101011) | ldf r2, 0x2b # Read from 32 bits from MD with prefetch
0005 5a01 (0101101000000001) | st r2, (r1, 0) # Address in r1
0006 1901 (0001100100000001) | addi r1, 1
| postloop:
0007 0300 (0000001100000000) | done 3
0008 0b00 (0000101100000000) | ldi r3, 0
0009 4b00 (0100101100000000) | cmpeqi r3, 0 # Always true
000a 7df5 (0111110111110101) | bt start # Always branches
------------ CUT HERE -----------
static const int sdma_code_length = 6;
static const u32 sdma_code[6] = {
0x6c20008f, 0x018e7803, 0x622b5a01, 0x19010300, 0x0b004b00, 0x7df50000,
};
Note that the arguments for sdf and ldf are given as numbers, and not following the not-so-helpful notation used in the Reference Manual.
The basic idea behind the assembly code is that each DW (Double Word, 32 bits) is read automatically by the functional unit from application space memory, and then fetched from the FIFO into r2. Then the register is written to SDMA memory with a plain “st” opcode.
The relevant tryrun() function to test this is:
static int tryrun(struct sdma_engine *sdma)
{
dma_addr_t src_phys;
void *src_virt;
const int channel = 1;
struct sdma_channel *sdmac = &sdma->channel[channel];
static const u32 sdma_code[6] = {
0x6c20008f, 0x018e7803, 0x622b5a01, 0x19010300, 0x0b004b00, 0x7df50000,
};
static const u32 sample_data[8] = {
0x12345678, 0x11223344, 0xdeadbeef, 0xbabecafe,
0xebeb0000, 0, 0xffffffff, 0xabcdef00 };
const int origin = 0xe00; // In data space terms (32 bits/address)
struct sdma_context_data *context = sdma->context;
int ret;
src_virt = dma_alloc_coherent(NULL,
4096, // 4096 bytes, just any buffer size
&src_phys, GFP_KERNEL);
if (!src_virt) {
printk(KERN_ERR "Failed to allocate source buffer memory\n");
return -ENOMEM;
}
memset(src_virt, 0, 4096);
memcpy(src_virt, sample_data, sizeof(sample_data));
sdma_write_datamem(sdma, (void *) sdma_code, sizeof(sdma_code), origin);
ret = sdma_request_channel(sdmac);
if (ret) {
printk(KERN_ERR "Failed to request channel\n");
return ret;
}
sdma_disable_channel(sdmac);
sdma_config_ownership(sdmac, false, true, false);
memset(context, 0, sizeof(*context));
context->channel_state.pc = origin * 2; // In program space addressing...
context->gReg[4] = src_phys;
context->gReg[6] = 0xe80;
context->gReg[7] = 3; // Number of DWs to copy
ret = sdma_write_datamem(sdma, (void *) context, sizeof(*context),
0x800 + (sizeof(*context) / 4) * channel);
if (ret) {
printk(KERN_ERR "Failed to load context\n");
return ret;
}
ret = sdma_run_channel(&sdma->channel[1]);
sdma_print_mem(sdma, 0xe80, 128);
if (ret) {
printk(KERN_ERR "Failed to run script!\n");
return ret;
}
return 0; /* Success! */
}
Note that the C code snippet, which is part of the output of the assembler compilation, actually appears in the tryrun() function.
Fast memcpy()
Assembly goes
$ ./sdma_asm.pl copydma.asm
| # Should be set up at invocation
| #
| # r0 : Number of DWs to copy (is altered as script runs)
| # r1 : Source address (DW aligned)
| # r2 : Destination address (DW aligned)
|
0000 6920 (0110100100100000) | stf r1, 0x20 # To MSA, prefetch on, address is nonfrozen
0001 6a04 (0110101000000100) | stf r2, 0x04 # To MDA, address is nonfrozen
0002 0c08 (0000110000001000) | ldi r4, 8 # Number of DWs to copy each round
| copyloop:
0003 04d8 (0000010011011000) | cmphs r4, r0 # Is 8 larger or equal to the number of DWs left to copy?
0004 7d03 (0111110100000011) | bt lastcopy # If so, jump to last transfer label
0005 6c18 (0110110000011000) | stf r4, 0x18 # Copy 8 words from MSA to MDA address.
0006 2008 (0010000000001000) | subi r0, 8 # Decrement counter
0007 7cfb (0111110011111011) | bf copyloop # Always branches, because r0 > 0
| lastcopy:
0008 6818 (0110100000011000) | stf r0, 0x18 # Copy 8 or less DWs (r0 is always > 0)
| exit:
0009 0300 (0000001100000000) | done 3
000a 0b00 (0000101100000000) | ldi r3, 0
000b 4b00 (0100101100000000) | cmpeqi r3, 0 # Always true
000c 7dfc (0111110111111100) | bt exit # Endless loop, just to be safe
------------ CUT HERE -----------
static const int sdma_code_length = 7;
static const u32 sdma_code[7] = {
0x69206a04, 0x0c0804d8, 0x7d036c18, 0x20087cfb, 0x68180300, 0x0b004b00, 0x7dfc0000,
}
For a frozen (constant) source address (e.g. when reading from a FIFO) the first stf should be done with argument 0x30 rather than 0x20. For a frozen destination address, the seconds stf has the argument 0x14 instead of 0x04.
This script should be started with r0 > 0. It may be OK to have r0=0, but I’m not sure about that (and if there’s no issue with not reading any data after a prefetch, as possibly related to section 52.22.1 in the Reference Manual).
The endless loop to “exit” should never be needed. It’s there just in case the script is rerun by mistake, so it responds with a “done” right away. And the example above is not really optimal: To make a for-sure branch, I could have gone “bt exit” and “bf exit” immediately after it, making this in two opcodes instead of three. Wasteful me.
The tryrun() function for this case then goes
static int tryrun(struct sdma_engine *sdma)
{
dma_addr_t buf_phys;
u8 *buf_virt;
const int channel = 1;
struct sdma_channel *sdmac = &sdma->channel[channel];
static const u32 sdma_code[7] = {
0x69206a04, 0x0c0804d8, 0x7d036c18, 0x20087cfb, 0x68180300, 0x0b004b00, 0x7dfc0000,
};
static const u32 sample_data[8] = {
0x12345678, 0x11223344, 0xdeadbeef, 0xbabecafe,
0xebeb0000, 0, 0xffffffff, 0xabcdef00 };
const int origin = 0xe00; // In data space terms (32 bits/address)
struct sdma_context_data *context = sdma->context;
int ret;
buf_virt = dma_alloc_coherent(NULL, 4096,
&buf_phys, GFP_KERNEL);
if (!buf_virt) {
printk(KERN_ERR "Failed to allocate source buffer memory\n");
return -ENOMEM;
}
memset(buf_virt, 0, 4096);
memcpy(buf_virt, sample_data, sizeof(sample_data));
sdma_write_datamem(sdma, (void *) sdma_code, sizeof(sdma_code), origin);
ret = sdma_request_channel(sdmac);
if (ret) {
printk(KERN_ERR "Failed to request channel\n");
return ret;
}
sdma_disable_channel(sdmac);
sdma_config_ownership(sdmac, false, true, false);
memset(context, 0, sizeof(*context));
context->channel_state.pc = origin * 2; // In program space addressing...
context->gReg[0] = 18; // Number of DWs to copy
context->gReg[1] = buf_phys;
context->gReg[2] = buf_phys + 0x40;
ret = sdma_write_datamem(sdma, (void *) context, sizeof(*context),
0x800 + (sizeof(*context) / 4) * channel);
if (ret) {
printk(KERN_ERR "Failed to load context\n");
return ret;
}
ret = sdma_run_channel(&sdma->channel[1]);
do {
int i;
const int len = 0xa0;
unsigned char line[128];
int pos = 0;
for (i=0; i<len; i++) {
if ((i % 16) == 0)
pos = sprintf(line, "%04x ", i);
pos += sprintf(&line[pos], "%02x ", buf_virt[i]);
if ((i % 16) == 15)
printk(KERN_WARNING "%s\n", line);
}
} while (0);
if (ret) {
printk(KERN_ERR "Failed to run script!\n");
return ret;
}
return 0; /* Success! */
}
The memory’s content is printed out here from tryrun() directly, since the dumped memory is in application space.