MySQL, OOM killer, overcommitting and other memory related issues

It started with an error message

This post is a bit of a coredump of myself attempting to resolve a sudden web server failure. And even more important, understand why it happened (check on that) and try avoiding it from happening in the future (not as lucky there).

I’ve noticed that there are many threads in the Internet on why mysqld died suddenly, so to make a long story short: mysqld has the exact profile that the OOM killer is looking for: Lots of resident RAM, and it’s not a system process. Apache gets killed every now and then for the same reason.

This post relates to a  VPS hosted Debian 8, kernel 3.10.0, x86_64. The MySQL server is a 5.5.62-0+deb8u1 (Debian).

As always, it started with a mail notification from some cronjob complaining about something. Soon enough it was evident that the MySQL server was down. And as usual, the deeper I investigated this issue, the more I realized that this was just the tip of the iceberg (the kind that doesn’t melt due to global warming).

The crash

So first, it was clear that the MySQL had restarted itself a couple of days before disaster:

191007  9:25:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
191007  9:25:17 [Note] Plugin 'FEDERATED' is disabled.
191007  9:25:17 InnoDB: The InnoDB memory heap is disabled
191007  9:25:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191007  9:25:17 InnoDB: Compressed tables use zlib 1.2.8
191007  9:25:17 InnoDB: Using Linux native AIO
191007  9:25:17 InnoDB: Initializing buffer pool, size = 128.0M
191007  9:25:17 InnoDB: Completed initialization of buffer pool
191007  9:25:17 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
191007  9:25:17  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
191007  9:25:19  InnoDB: Waiting for the background threads to start
191007  9:25:20 InnoDB: 5.5.62 started; log sequence number 1427184442
191007  9:25:20 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
191007  9:25:20 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
191007  9:25:20 [Note] Server socket created on IP: '127.0.0.1'.
191007  9:25:21 [Note] Event Scheduler: Loaded 0 events
191007  9:25:21 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.62-0+deb8u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_options' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_options'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_posts' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_posts'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_taxonomy' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_term_taxonomy'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_relationships' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_term_relationships'

And then, two days layer, it crashed for real. Or actually, got killed. From the syslog:

Oct 09 05:30:16 kernel: OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

and

191009  5:30:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
191009  5:30:17 [Note] Plugin 'FEDERATED' is disabled.
191009  5:30:17 InnoDB: The InnoDB memory heap is disabled
191009  5:30:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191009  5:30:17 InnoDB: Compressed tables use zlib 1.2.8
191009  5:30:17 InnoDB: Using Linux native AIO
191009  5:30:17 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
191009  5:30:17 InnoDB: Completed initialization of buffer pool
191009  5:30:17 InnoDB: Fatal error: cannot allocate memory for the buffer pool
191009  5:30:17 [ERROR] Plugin 'InnoDB' init function returned error.
191009  5:30:17 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
191009  5:30:17 [ERROR] Unknown/unsupported storage engine: InnoDB
191009  5:30:17 [ERROR] Aborting

191009  5:30:17 [Note] /usr/sbin/mysqld: Shutdown complete

The mmap() is most likely anonymous (i.e. not related to a file), as I couldn’t find any memory mapped file that is related to the mysql processes (except for the obvious mappings of shared libraries).

The smoking gun

But here comes the good part: It turns out that the OOM killer had been active several times before. It just so happen that the processes are being newborn every time this happens. It was the relaunch that failed this time — otherwise I wouldn’t have noticed this was going on.

This is the output of plain “dmesg”. All OOM entries but the last one were not available with journalctl, as old entries had been deleted.

[3634197.152028] OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB
[3634197.273914] OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB
[4487991.904510] OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB
[4835006.413510] OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB
[4835006.767112] OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB
[4884915.371805] OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB
[4884915.509686] OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB
[5096265.088151] OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB
[5845437.591031] OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB
[5845437.608682] OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB
[6896254.741732] OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB
[7054957.856153] OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

Or, after calculating the time stamps (using the last OOM message as a reference):

Fri Aug 30 15:17:36 2019 OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB
Fri Aug 30 15:17:36 2019 OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB
Mon Sep  9 12:27:30 2019 OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB
Fri Sep 13 12:51:05 2019 OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB
Fri Sep 13 12:51:05 2019 OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB
Sat Sep 14 02:42:54 2019 OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB
Sat Sep 14 02:42:54 2019 OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB
Mon Sep 16 13:25:24 2019 OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB
Wed Sep 25 05:31:36 2019 OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB
Wed Sep 25 05:31:36 2019 OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB
Mon Oct  7 09:25:13 2019 OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB
Wed Oct  9 05:30:16 2019 OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

anon-rss is the resident RAM consumed by the process itself (anonymous = not memory mapped to a file or something like that).

total-vm is the total size of the Virtual Memory in use. This isn’t very relevant (I think), as it involves shared libraries, memory mapped files and other segments that don’t consume any actual RAM or other valuable resources.

So now it’s clear what happened. Next, to some finer resolution.

The MySQL keepaliver

The MySQL daemon is executed by virtue of an SysV init script, which launches /usr/bin/mysqld_safe, a patch-on-patch script to keep the daemon alive, no matter what. It restarts the mysqld daemon if it dies for any or no reason, and should also produce log messages. On my system, it’s executed as

/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

The script issues log messages when something unexpected happens, but they don’t appear in /var/log/mysql/error.log or anywhere else, even though the file exists, is owned by the mysql user, and has quite a few messages from the mysql daemon itself.

Changing

/usr/bin/mysqld_safe > /dev/null 2>&1 &

to

/usr/bin/mysqld_safe --syslog > /dev/null 2>&1 &

Frankly speaking, I don’t think this made any difference. I’ve seen nothing new in the logs.

It would have been nicer having the messages in mysql/error.log, but at least they are visible with journalctl this way.

Shrinking the InnoDB buffer pool

As the actual failure was on attempting to map memory for the buffer pool, maybe make it smaller…?

Launch MySQL as the root user:

$ mysql -u root --password

and check the InnoDB status, as suggested on this page:

mysql> SHOW ENGINE INNODB STATUS;

[ ... ]

----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 137363456; in additional pool allocated 0
Dictionary memory allocated 1100748
Buffer pool size   8192
Free buffers       6263
Database pages     1912
Old database pages 725
Modified db pages  0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1894, created 18, written 1013
0.00 reads/s, 0.00 creates/s, 0.26 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1912, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]

I’m really not an expert, but if “Free buffers” is 75% of the total allocated space, I’ve probably allocated too much. So I reduced it to 32 MB — it’s not like I’m running a high-end server. I added /etc/mysql/conf.d/innodb_pool_size.cnf (owned by root, 0644) reading:

# Reduce InnoDB buffer size from default 128 MB to 32 MB
[mysqld]
innodb_buffer_pool_size=32M

Restarting the daemon, it says:

----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 34340864; in additional pool allocated 0
Dictionary memory allocated 1100748
Buffer pool size   2047
Free buffers       856
Database pages     1189
Old database pages 458

And finally, repair the tables

Remember those warnings that the tables were marked as crashed? That’s the easy part:

$ mysqlcheck -A --auto-repair

That went smoothly, with no complaints. After all, it wasn’t really a crash.

Some general words on OOM

This whole idea that the kernel should do Roman Empire style decimation of processes is widely criticized by many, but it’s probably not such a bad idea. The root cause lies in the fact that the kernel agrees to allocate more RAM than it actually has. This is even possible because the kernel doesn’t really allocate RAM when a process asks for memory with a brk() call, but it only allocates the memory space segment. The actual RAM is allocated only when the process attempts to access a page that hasn’t been RAM allocated yet. The access attempt causes a page fault, the kernel quickly fixes some RAM and returns from the page fault interrupt as if nothing happened.

So when the kernel responds with an -ENOMEM, it’s not because it doesn’t have any RAM, but because it doesn’t want to.

More precisely, the kernel keeps account on how much memory it has given away (system-wise and/or cgroup-wise) and make a decision. The common policy is to overcommit to some extent — that is, to allow the total allocated RAM allocated to exceed the total physical RAM. Even, and in particular, if there’s no swap.

The common figure is to overcommit by 50%: For a 64 GiB RAM computer, there might be 96 GiB or promised RAM. This may seem awfully stupid thing to do, but hey, it works. If that concept worries you, modern banking (with real money, that is) might worry you even more.

The problem rises when the processes run to the bank. That is, when the processes access the RAM they’ve been promised, and at some point the kernel has nowhere to take memory from. Let’s assume there’s no swap, all disk buffers have been flushed, all rabbits have been pulled. There’s a process waiting for memory, and it can’t go back running until the problem has been resolved.

Linux’ solution to this situation is to select a process with a lot of RAM and little importance. How the kernel does that judgement is documented everywhere. The important point is that it’s not necessarily the process that triggered the event, and that it will usually be the same victim over and over again. In my case, mysqld is the favorite. Big, fat, and not a system process.

Thinking about it, the OOM is a good solution to get out of a tricky situation. The alternative would have been to deny memory to processes just launched, including the administrator’s attempt to rescue the system. Or an attempt to shut it down with some dignity. So sacrificing a large and hopefully not-so-important process isn’t such a bad idea.

Why did the OOM kick in?

This all took place on a VPS virtual machine with 1 GB leased RAM. With the stuff running on that machine, there’s no reason in the world that the total actual RAM consumption would reach that limit. This is a system that typically has 70% of its memory marked as “cached” (i.e. used by disk cache). This should be taken with a grain of salt, as “top” displays data from some bogus /proc/vmstat, and still.

As can be seen in the dmesg logs above, the amount of resident RAM of the killed mysqld process was 120-150 MB or so. Together with the other memory hog, apache2, they reach 300 MB. That’s it. No reason for anything drastic.

Having said that, it’s remarkable that the total-vm stood at 2.4-4.3 GB when killed. This is much higher than the typical 900 MB visible usually. So maybe there’s some kind of memory leak, even if it’s harmless? Looking at mysql over time, its virtual memory allocation tends to grow.

VPS machines do have a physical memory limit imposed, by virtue of the relevant cgroup’s memory.high and memory.max limits. In particular the latter — if the cgroup’s total consumption exceeds memory.max, OOM kicks in. This is how the illusion of an independent RAM segment is made on a VPS machine. Plus faking some /proc files.

But there’s another explanation: Say that a VPS service provider takes a computer with 16 GB RAM, and places 16 VPS machines with 1 GB leased RAM each. What will the overall actual RAM consumption be? I would expect it to be much lower than 16 GB. So why not add a few more VPS machines, and make some good use of the hardware? It’s where the profit comes from.

Most of the time, there will be no problem. But occasionally, this will cause RAM shortages, in which case the kernel’s global OOM looks for a victim. I suppose there’s no significance to cgroups in this matter. In other words, the kernel sees all processes in the system the same, regardless of which cgroup (and hence VPS machine) they belong to. Which means that the process killed doesn’t necessarily belong to the VPS that triggered the problem. The processes of one VPS may suddenly demand their memory, but some other VPS will have its processes killed.

Conclusion

  • Shrinking the buffer pool of mysqld was probably a good idea, in particular if a computer-wide OOM killed the process — odds are that it will kill some other mysqld instead this way.
  • Possibly restart mysql with a cronjob every day to keep its memory consumption in control. But this might create problems of its own.
  • It’s high time to replace the VPS guest with KVM or similar.

—————————————————————-

Rambling epilogue: Some thoughts about overcomitting

The details for how overcomitting is accounted for is given on the kernel tree’s Documentation/vm/overcommit-accounting. But to make a long story short, it’s done in a sensible way. In particular, if a piece of memory is shared by threads and processes, it’s only accounted for once.

Relevant files: /proc/meminfo and /proc/vmstat

It seems like CommitLimit and Committed_AS are not available on a VPS guest system. But the OOM killer probably knows these values (or was it because /proc/sys/vm/overcommit_memory was set to 1 on my system, meaning “Always overcommit”?).

To get a list of the current memory hogs, run “top” and press shift-M as it’s running.

To get an idea on how a process behaves, use pmap -x. For example, looking at a mysqld process (run as root, or no memory map will be shown):

# pmap -x 14817
14817:   /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
Address           Kbytes     RSS   Dirty Mode  Mapping
000055c5617ac000   10476    6204       0 r-x-- mysqld
000055c5623e6000     452     452     452 r---- mysqld
000055c562457000     668     412     284 rw--- mysqld
000055c5624fe000     172     172     172 rw---   [ anon ]
000055c563e9b000    6592    6448    6448 rw---   [ anon ]
00007f819c000000    2296     320     320 rw---   [ anon ]
00007f819c23e000   63240       0       0 -----   [ anon ]
00007f81a0000000    3160     608     608 rw---   [ anon ]
00007f81a0316000   62376       0       0 -----   [ anon ]
00007f81a4000000    9688    7220    7220 rw---   [ anon ]
00007f81a4976000   55848       0       0 -----   [ anon ]
00007f81a8000000     132       8       8 rw---   [ anon ]
00007f81a8021000   65404       0       0 -----   [ anon ]
00007f81ac000000     132       4       4 rw---   [ anon ]
00007f81ac021000   65404       0       0 -----   [ anon ]
00007f81b1220000       4       0       0 -----   [ anon ]
00007f81b1221000    8192       8       8 rw---   [ anon ]
00007f81b1a21000       4       0       0 -----   [ anon ]
00007f81b1a22000    8192       8       8 rw---   [ anon ]
00007f81b2222000       4       0       0 -----   [ anon ]
00007f81b2223000    8192       8       8 rw---   [ anon ]
00007f81b2a23000       4       0       0 -----   [ anon ]
00007f81b2a24000    8192      20      20 rw---   [ anon ]
00007f81b3224000       4       0       0 -----   [ anon ]
00007f81b3225000    8192       8       8 rw---   [ anon ]
00007f81b3a25000       4       0       0 -----   [ anon ]
00007f81b3a26000    8192       8       8 rw---   [ anon ]
00007f81b4226000       4       0       0 -----   [ anon ]
00007f81b4227000    8192       8       8 rw---   [ anon ]
00007f81b4a27000       4       0       0 -----   [ anon ]
00007f81b4a28000    8192       8       8 rw---   [ anon ]
00007f81b5228000       4       0       0 -----   [ anon ]
00007f81b5229000    8192       8       8 rw---   [ anon ]
00007f81b5a29000       4       0       0 -----   [ anon ]
00007f81b5a2a000    8192       8       8 rw---   [ anon ]
00007f81b622a000       4       0       0 -----   [ anon ]
00007f81b622b000    8192      12      12 rw---   [ anon ]
00007f81b6a2b000       4       0       0 -----   [ anon ]
00007f81b6a2c000    8192       8       8 rw---   [ anon ]
00007f81b722c000       4       0       0 -----   [ anon ]
00007f81b722d000   79692   57740   57740 rw---   [ anon ]
00007f81bc000000     132      76      76 rw---   [ anon ]
00007f81bc021000   65404       0       0 -----   [ anon ]
00007f81c002f000    2068    2052    2052 rw---   [ anon ]
00007f81c03f9000       4       0       0 -----   [ anon ]
00007f81c03fa000     192      52      52 rw---   [ anon ]
00007f81c042a000       4       0       0 -----   [ anon ]
00007f81c042b000     192      52      52 rw---   [ anon ]
00007f81c045b000       4       0       0 -----   [ anon ]
00007f81c045c000     192      64      64 rw---   [ anon ]
00007f81c048c000       4       0       0 -----   [ anon ]
00007f81c048d000     736     552     552 rw---   [ anon ]
00007f81c0545000      20       4       0 rw-s- [aio] (deleted)
00007f81c054a000      20       4       0 rw-s- [aio] (deleted)
00007f81c054f000    3364    3364    3364 rw---   [ anon ]
00007f81c0898000      44      12       0 r-x-- libnss_files-2.19.so
00007f81c08a3000    2044       0       0 ----- libnss_files-2.19.so
00007f81c0aa2000       4       4       4 r---- libnss_files-2.19.so
00007f81c0aa3000       4       4       4 rw--- libnss_files-2.19.so
00007f81c0aa4000      40      20       0 r-x-- libnss_nis-2.19.so
00007f81c0aae000    2044       0       0 ----- libnss_nis-2.19.so
00007f81c0cad000       4       4       4 r---- libnss_nis-2.19.so
00007f81c0cae000       4       4       4 rw--- libnss_nis-2.19.so
00007f81c0caf000      28      20       0 r-x-- libnss_compat-2.19.so
00007f81c0cb6000    2044       0       0 ----- libnss_compat-2.19.so
00007f81c0eb5000       4       4       4 r---- libnss_compat-2.19.so
00007f81c0eb6000       4       4       4 rw--- libnss_compat-2.19.so
00007f81c0eb7000       4       0       0 -----   [ anon ]
00007f81c0eb8000    8192       8       8 rw---   [ anon ]
00007f81c16b8000      84      20       0 r-x-- libnsl-2.19.so
00007f81c16cd000    2044       0       0 ----- libnsl-2.19.so
00007f81c18cc000       4       4       4 r---- libnsl-2.19.so
00007f81c18cd000       4       4       4 rw--- libnsl-2.19.so
00007f81c18ce000       8       0       0 rw---   [ anon ]
00007f81c18d0000    1668     656       0 r-x-- libc-2.19.so
00007f81c1a71000    2048       0       0 ----- libc-2.19.so
00007f81c1c71000      16      16      16 r---- libc-2.19.so
00007f81c1c75000       8       8       8 rw--- libc-2.19.so
00007f81c1c77000      16      16      16 rw---   [ anon ]
00007f81c1c7b000      88      44       0 r-x-- libgcc_s.so.1
00007f81c1c91000    2044       0       0 ----- libgcc_s.so.1
00007f81c1e90000       4       4       4 rw--- libgcc_s.so.1
00007f81c1e91000    1024     128       0 r-x-- libm-2.19.so
00007f81c1f91000    2044       0       0 ----- libm-2.19.so
00007f81c2190000       4       4       4 r---- libm-2.19.so
00007f81c2191000       4       4       4 rw--- libm-2.19.so
00007f81c2192000     944     368       0 r-x-- libstdc++.so.6.0.20
00007f81c227e000    2048       0       0 ----- libstdc++.so.6.0.20
00007f81c247e000      32      32      32 r---- libstdc++.so.6.0.20
00007f81c2486000       8       8       8 rw--- libstdc++.so.6.0.20
00007f81c2488000      84       8       8 rw---   [ anon ]
00007f81c249d000      12       8       0 r-x-- libdl-2.19.so
00007f81c24a0000    2044       0       0 ----- libdl-2.19.so
00007f81c269f000       4       4       4 r---- libdl-2.19.so
00007f81c26a0000       4       4       4 rw--- libdl-2.19.so
00007f81c26a1000      32       4       0 r-x-- libcrypt-2.19.so
00007f81c26a9000    2044       0       0 ----- libcrypt-2.19.so
00007f81c28a8000       4       4       4 r---- libcrypt-2.19.so
00007f81c28a9000       4       4       4 rw--- libcrypt-2.19.so
00007f81c28aa000     184       0       0 rw---   [ anon ]
00007f81c28d8000      36      28       0 r-x-- libwrap.so.0.7.6
00007f81c28e1000    2044       0       0 ----- libwrap.so.0.7.6
00007f81c2ae0000       4       4       4 r---- libwrap.so.0.7.6
00007f81c2ae1000       4       4       4 rw--- libwrap.so.0.7.6
00007f81c2ae2000       4       4       4 rw---   [ anon ]
00007f81c2ae3000     104      12       0 r-x-- libz.so.1.2.8
00007f81c2afd000    2044       0       0 ----- libz.so.1.2.8
00007f81c2cfc000       4       4       4 r---- libz.so.1.2.8
00007f81c2cfd000       4       4       4 rw--- libz.so.1.2.8
00007f81c2cfe000       4       4       0 r-x-- libaio.so.1.0.1
00007f81c2cff000    2044       0       0 ----- libaio.so.1.0.1
00007f81c2efe000       4       4       4 r---- libaio.so.1.0.1
00007f81c2eff000       4       4       4 rw--- libaio.so.1.0.1
00007f81c2f00000      96      84       0 r-x-- libpthread-2.19.so
00007f81c2f18000    2044       0       0 ----- libpthread-2.19.so
00007f81c3117000       4       4       4 r---- libpthread-2.19.so
00007f81c3118000       4       4       4 rw--- libpthread-2.19.so
00007f81c3119000      16       4       4 rw---   [ anon ]
00007f81c311d000     132     112       0 r-x-- ld-2.19.so
00007f81c313e000       8       0       0 rw---   [ anon ]
00007f81c3140000      20       4       0 rw-s- [aio] (deleted)
00007f81c3145000      20       4       0 rw-s- [aio] (deleted)
00007f81c314a000      20       4       0 rw-s- [aio] (deleted)
00007f81c314f000      20       4       0 rw-s- [aio] (deleted)
00007f81c3154000      20       4       0 rw-s- [aio] (deleted)
00007f81c3159000      20       4       0 rw-s- [aio] (deleted)
00007f81c315e000      20       4       0 rw-s- [aio] (deleted)
00007f81c3163000      20       4       0 rw-s- [aio] (deleted)
00007f81c3168000    1840    1840    1840 rw---   [ anon ]
00007f81c3334000       8       0       0 rw-s- [aio] (deleted)
00007f81c3336000       4       0       0 rw-s- [aio] (deleted)
00007f81c3337000      24      12      12 rw---   [ anon ]
00007f81c333d000       4       4       4 r---- ld-2.19.so
00007f81c333e000       4       4       4 rw--- ld-2.19.so
00007f81c333f000       4       4       4 rw---   [ anon ]
00007ffd2d68b000     132      68      68 rw---   [ stack ]
00007ffd2d7ad000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB          640460   89604   81708

The KBytes and RSS column’s Total at the bottom matches the VIRT and RSS figures shown by “top”.

I should emphasize that this a freshly started mysqld process. Give it a few days to run, and some extra 100 MB of virtual space is added (not clear why) plus some real RAM, depending on the setting.

I’ve marked six anonymous segments that are completely virtual (no resident memory at all) summing up to ~360 MB. This means that they are counted in as 360 MB at least once — and that’s for a process that only uses 90 MB for real.

My own anecdotal test on another machine with a 4.4.0 kernel showed that putting /proc/sys/vm/overcommit_ratio below what was actually committed (making /proc/meminfo’s CommitLimit smaller than Committed_AS) didn’t have any effect unless /proc/sys/vm/overcommit_memory was set to 2. And when I did that, the OOM wasn’t called, but instead I had a hard time running new commands:

# echo 2 > /proc/sys/vm/overcommit_memory
# cat /proc/meminfo
bash: fork: Cannot allocate memory

So this is what it looks like when memory runs out and the system refuses to play ball.

AudioCodes MP202B hacking notes (Netvision)

This is a no-go

These are notes during my failed attempts to turn an old (2011?) Israeli ISP provider’s SIP adapter (MP202B, provided by Netvision) into one that works with any SIP provider. The first step is to get access of its configuration, and that’s exactly where it stopped.

Netvision has taken action to prevent any manipulations, in particular by keeping the access password secret (or maybe there is no password at all?). Some will call it customer retention.

So here is my war story. Sometimes you win, sometimes you lose.

Trying the nice way

Plugged the adapter’s LAN Ethernet jack to my laptop through the USB Ethernet adapter. The adapter served as a DHCP server, and gave the laptop the address 10.100.102.2. The default gateway as well as nameserver were set to 10.100.102.1.

So far so good. I opened the browser, and attempted to open the http://mp202.home, as the manual says. The browser complained that page can’t be opened, and indeed, a DNS query was made, giving the A address records 10.100.102.1 and 10.192.0.1 for that domain, however the attempt to open a TCP connection on port 80 of either of these addresses was responded with a flat TCP RST.

But it does work with http://mp202.home:8080 instead.

Telnet also worked, prompting for user and password.

However I failed to login, both from telnet and via the web interface. The default admin/admin pair didn’t work, and neither did the user/wel-012! pair mentioned a lot in forums, specifically for the adapter from Netvision. Word has it in old forums that Netvision changes the password all the time to prevent users from fiddling with their equipment. Peculiarly, when typing “user” at telnet’s prompt, it doesn’t ask for password, but goes on asking for username again.

So reset to factory defaults. The instructions in the Manual were to hold the RST pin for 7 seconds and up, but that did nothing. So I powered the device off, disconnected the network cable, and repowered with the RST pin held for a few seconds. And that clearly made a difference — in particular, the device stopped attempting to contact the SIP server as it did before.

Also: The LAN server now answered to port 80 for http://mp202.home.

But none of the user/password combinations I could think about worked.

Opening the MP202B

The screws are under the small rubber pieces that hold the adapter, four of them. There’s just a small PCB, with one interesting pin header, labeled JP1. Three pins. As this is obviously a Linux system, there’s almost a UART header. And this is the case.

So JP1 is UART, 3.3V, plain 115200 8N1. Plain USB adapters work. Connection:

  • Pin 1: Tx (driven by adapter). White wire
  • Pin 2: GND
  • Pin 3: Rx (pulled up by device). Green wire.

For some reason, the adapter didn’t boot up when pin 3 was connected to the green wire. So disconnect, let it boot (the boot messages are visible), connect the green wire and reboot by a quick power cycle.

As one can guess from the introduction, this didn’t help much either: Except for a lot of boot messages (see below), the serial port requires a login, exactly the same as the telnet.

The one thing I could possibly have done is to tftp an image of the mounted file system to my computer, mount it there, change the /etc/passwd file and then write it back to the flash. Maybe that would have worked.

However this is where it stopped to be fun, and my time is worth more than that.

Speaking of fun, a serial port session log follows. Note how clean and elegant the boot messages are. Or: I don’t know why it always ends up like this on closed hood systems.

Session log

Flash found: MX29LV640DBTC
Press ESC for monitor... 22
(ac494bl) help
reboot            version           info              fa
printenv          setenv            unsetenv          defragenv
fmt               boot              boot_recovery     dm
oclk              help              tftp              memtest           

(ac494bl) boot_recovery

Booting...
Uncompressing Linux.................................................................................................................Ok
Booting the kernel at 0x94020790

LINUX started...
CPU revision is: 00018448
Primary instruction cache 16kb, linesize 16 bytes (4 ways)
Primary data cache 16kb, linesize 16 bytes (4 ways)
Linux version 2.4.21openrg-rmk1 #5 Mon Sep 7 13:40:37 IDT 2009

 leds re-enabled @@@

Determined physical RAM map:
 memory: 14000000 @ 00000000 (reserved)
 memory: 00020000 @ 14000000 (ROM data)
 memory: 00f60000 @ 14020000 (usable)
On node 0 totalpages: 3968
zone(0): 3968 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: console=ttyS0,115200 root=/dev/ram0 rw
calculating r4koff... 00098968(625000)
CPU frequency 125.00 MHz
Calibrating delay loop... 124.92 BogoMIPS
Freeing Adam2 reserved memory [0x14001000,0x0001f000]
Memory: 11972k/15872k available (1131k kernel code, 3900k reserved, 2388k data, 4k init)
Dentry cache hash table entries: 2048 (order: 2, 16384 bytes)
Inode cache hash table entries: 1024 (order: 1, 8192 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 4096 (order: 2, 16384 bytes)
Checking for 'wait' instruction...  available.
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd

Random: 0x323bdaca
pty: 256 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with no serial options enabled
ttyS00 at 0xa8610e00 (irq = 15) is a 16550A
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 512 bind 1024)
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
VFS: Mounted root (tmpfs filesystem).
Mount of initial cramfs successful
Freeing init cramfs memory: 64k freed
Freeing unused kernel memory: 4k freed
Mounting cramfs image at 'cramfs'
Algorithmics/MIPS FPU Emulator v1.5
insmod: add-symbol-file PATH/kos_lib.o 0xc0000060 -s .data 0xc000ddc0 -s .bss 0xc000eb90
insmod: add-symbol-file PATH/avaavalanche flash device: 0x800000 at 0x10000000.
lanche_flash_mod-----> [jedec_probe_chip:669]mfr=0x00C2, id=0x22CB, base=0x0000, osf=0x0002.
.o 0xc0011060 -sFound: Macronix MX29LV640DBTC
 .data 0xc001196Physically mapped flash:0: Found 1 x16 devices at 0x0 in 16-bit mode
0 -s .bss 0xc001 Amd/Fujitsu Extended Query Table vÿ.ÿ at 0x0000
1a80
number of CFI chips: 1
avalanche flash device: 0x800000 at 0x1d000000.
-----> [jedec_probe_chip:669]mfr=0x0000, id=0x0000, base=0x0000, osf=0x0002.
-----> [jedec_probe_chip:669]mfr=0x0000, id=0x0000, base=0x0000, osf=0x0002.
-----> [jedec_probe_chip:669]mfr=0x9090, id=0x9090, base=0x0000, osf=0x0002.
-----> [jedec_probe_chip:669]mfr=0x9090, id=0x9090, base=0x0000, osf=0x0002.
-----> [jedec_probe_chip:669]mfr=0x9090, id=0x9090, base=0x0000, osf=0x0004.
-----> [jedec_probe_chip:669]mfr=0x9090, id=0x9090, base=0x0000, osf=0x0004.
JEDEC: Found no Physically mapped flash:1 device at location zero
registering mtd failed
failed initializing  flash dev 1
Looking for mtd device :mtd0:
Found a mtd0 image (0x10000), with size (0x560000).
Creating 1 MTD partitions on "Physically mapped flash:0":
0x00010000-0x00570000 : "mtd0"
Looking for mtd device :mtd1:
insmod: add-symbol-file PATH/avalanche_cpmac.o 0xc0013060 -s .data 0xc0024240 -s .bss 0xc0024bf0
loading license key: AUDIOCODES
Press ESC to enter BOOT MENU mode.
[mt_rg_conf_compat:2333] saved_version=4.10.4.5.2, rg_version=4.1.7.3.7, saved_ext_version=3.0.1_p041_build_19, external_version=2.6.2_p10_build_2 .
[mt_rg_conf_compat:2335] UPDATING SERIAN NUMBER .....
ACL_GetSerialNumberFromMAC: mac_str = 00:90:8F:2F:9B:92
ACL_GetSerialNumberFromMAC: iserial_num=3120018 .
[mt_rg_conf_compat:2350] check_branch=-1!!!
[mt_rg_conf_compat:2353] restoring default configuration - new ACL branch !!!

Going to reboot!
Restarting system.

Flash found: MX29LV640DBTC
Press ESC for monitor... 221
(ac494bl) help
reboot            version           info              fa
printenv          setenv            unsetenv          defragenv
fmt               boot              boot_recovery     dm
oclk              help              tftp              memtest           

(ac494bl) version

PSPBoot 2.3.0.4
Compiled gcc rev: 3.3.2 [Aug  5 2008 12:29:56]
Built for MP202 board in Little Endian mode.

Optional modules included (+) or not (-):
 +tibinary +elf -gzip -ffs +tftp -ftp -dhcp -pcapp
(ac494bl) info

CHIP ID: TNETV1050 (0x7), REV: 0x2

MIPS Processor   : 4KEc rev: 2.2.0
Cache mode       : write-back, write-allocate.
Instruction cache: Associativity: 4, Line size: 16, Total size: 16KB
Data cache       : Associativity: 4, Line size: 16, Total size: 16KB

Last reset cause: Software reset (memory controller also reset)

EMIF is running at the same speed of the processor.
Processor running in little endian mode.
Processor clock is synchronous to internal bus clock.

(ac494bl) fa
Current Flash Allocation:

section :   PSBL, base : 0xb0000000, size :      50368 bytes
section :    ENV, base : 0xb07f0000, size :      65536 bytes

unallocated Space Start: 0xb0010000
unallocated Space End  : 0xb07f0000

(ac494bl) printenv

HWA_0        	00:01:02:03:04:05
HWA_ESWITCH  	00:90:8F:04:C0:8F
MAXLDRVER    	2.3.0.4
MODETTY0     	115200,n,8,1,hw
MODETTY1     	115200,n,8,1,hw
CPUFREQ      	125000000
SYSFREQ      	125000000
PROMPT       	(ac494bl)
mtd2         	0x90000000,0x90010000
mtd0         	0x90010000,0x90570000
mtd3         	0x907f0000,0x90800000
mtd4         	0x90570000,0x907f0000
BOOTRECOVERY 	m:f:"mtd4"
BOOTCFG      	m:f:"mtd0"
IPA          	192.168.1.1
IPA_GATEWAY  	192.168.1.254
SUBNET_MASK  	255.255.255.0
IPA_SVR      	192.168.1.2
MEMSZ        	0x00f80000
FLASHSZ      	0x00800000
BANNER       	1
FACTORY_DFT  	0

(ac494bl) boot
[boot 84] attempt (0) trying to boot from (BOOTCFG)

Booting...

sizeof(char): 00000001
sizeof(short): 00000002
sizeof(int): 00000004
sizeof(size_t): 00000004
sizeof(long): 00000004
sizeof(addr_t): 00000004
CONFIG_MTD_PHYSMAP_START: 90000000
CONFIG_MTD_PHYSMAP_LEN: 00800000
KERNEL_START: 94020000
virt_to_program(KERNEL_START): 94020000
HEAD_ADDR: 94C00000
CRAMFS_TARGET_ADDR: 943C2000
CRAMFS_LEN: 00300000
head_start_addr: 94C0000C
start_addr: 90010018
program_to_phys(start_addr): 90010018
offset: FB41000C
&_compressed_kernel_start: 94C02170
&_compressed_kernel_end: 94CDD602
compressed_kernel_start: 9001217C
compressed_kernel_len: 000DB492
Uncompressing Linux...
from: 9001217C
output_data: 94020000
output_len: 00BE0000
..........................................................................................................res: 00358000
Done
&_rofs_start: 94CDE000
&_rofs_end: 94FDE000
test: 90010018
_PA(test): 90010018
_PA(CONFIG_MTD_PHYSMAP_START): 90000000
_PA(CONFIG_MTD_PHYSMAP_START) + CONFIG_MTD_PHYSMAP_LEN: 90800000
test: 90010018
CramFS in Flash
Check CRC ... ||\-/|\-/|\-/177EC94A OK
cramfs_start_src: 00000000
CramFS dump (offsets in words):
+0: 28CD3D45
+1: 00300000
+2: 0000A003
kernel: 94020790
Kernel dump:
+0: 3C1C9415
+1: 279C2000
+2: 27881FE0
Booting the kernel

LINUX started...
rofs_addr: 900ee00c (100ee00c), len: 300000, flash_start_addr: 90000000 (10000000) flash_len: 800000
CramFS in Flash (prom)
rofs_start_addr: 00000000 len: 0
CPU revision is: 00018448
Primary instruction cache 16kB, physically tagged, 4-way, linesize 16 bytes.
Primary data cache 16kB 4-way, linesize 16 bytes.
Linux version 2.4.21openrg-rmk1 #122 Wed Dec 15 18:44:20 IST 2010

 leds re-enabled @@@

Determined physical RAM map:
 memory: 14000000 @ 00000000 (reserved)
 memory: 00020000 @ 14000000 (ROM data)
 memory: 00f60000 @ 14020000 (usable)
Using memsize of f80000
On node 0 totalpages: 3968
zone(0): 3968 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: cramfs=flash mtdparts=phys_mapped_flash:0x800000@0(Gateway),0x300000@0x000ee00c(mainfs) slram=initfs,0x14368000,+65536 console=ttyS0,115200 root=/dev/ram0 rw
calculating r4koff... 00098968(625000)
CPU frequency 125.00 MHz
Calibrating delay loop... 124.51 BogoMIPS
Freeing Adam2 reserved memory [0x14001000,0x0001f000]
Memory: 11964k/15872k available (1216k kernel code, 3908k reserved, 2184k data, 4k init)
Dentry cache hash table entries: 2048 (order: 2, 16384 bytes)
Inode cache hash table entries: 1024 (order: 1, 8192 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 4096 (order: 2, 16384 bytes)
Checking for 'wait' instruction...  available.
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
/proc/fs/cramfs_block_uncompressed created
/proc/fs/cramfs_location created

Random: 0x665ef687
pty: 256 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with no serial options enabled
ttyS00 at 0xa8610e00 (irq = 15) is a 16550A
AVALANCHE flash device: 800000 at 10000000
-----> [jedec_probe_chip:669]mfr=0x00C2, id=0x22CB, base=0x0000, osf=0x0002.
Found: Macronix MX29LV640DBTC
Avalanche Physically mapped flash: Found 1 x16 devices at 0x0 in 16-bit mode
 Amd/Fujitsu Extended Query Table vÿ.ÿ at 0x0000
number of CFI chips: 1
Using command line partition definition
Creating 2 MTD partitions on "Avalanche Physically mapped flash":
0x00000000-0x00800000 : "Gateway"
0x000ee00c-0x003ee00c : "mainfs"
mtd: partition "mainfs" doesn't start on an erase block boundary -- force read-only
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 512 bind 1024)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
VFS: Mounted root (tmpfs filesystem).
making node /dev/mtdblock3 (initfs)
Mount of initial cramfs successful
Error making node /dev/mtdblock2
making node /dev/mtdblock2 (mainfs)
Freeing unused kernel memory: 4k freed
AVALANCHE flash device: 800000 at 10000000
-----> [jedec_probe_chip:669]mfr=0x00C2, id=0x22CB, base=0x0000, osf=0x0002.
Found: Macronix MX29LV640DBTC
Avalanche Physically mapped flash: Found 1 x16 devices at 0x0 in 16-bit mode
 Amd/Fujitsu Extended Query Table vÿ.ÿ at 0x0000
number of CFI chips: 1
Using command line partition definition
Creating 2 MTD partitions on "Avalanche Physically mapped flash":
0x00000000-0x00800000 : "Gateway"
0x000ee00c-0x003ee00c : "mainfs"
mtd: partition "mainfs" doesn't start on an erase block boundary -- force read-only
Algorithmics/MIPS FPU Emulator v1.5
Version: 3.0.1_p041_build_19
Platform: AudioCodes MP20xB Board With SILABS Slic
Compilation Time: 15-Dec-10 16:38:49
Tag: Ntag-4_10_4_X_X
Compilation Flags: CONFIG_RG_VOIP_RV_MGCP=n CONFIG_RG_VOIP_RV_SIP=y BUILD=build.REV_B_SIP DIST=MP20XB_SILABS_VGW LIC=../jpkg_audiocodes.lic
User Information: audiocodes@localhost.localdomain /home/audiocodes/work/mp20x_4.10/amil_MP20x_4.10_dev6/CMBU/tulip/rg

insmod: add-symbol-file build/debug/one_module.o 0xc0011060 -s .main_flow 0xc00543e0 -s .data 0xc005f8b0 -s .bss 0xc0061760
Loading license 28fddd6dd30365cff7d05e4bb8a26f22278422545c33a022d98c810b0bb80b20e41b129a325365e4f9c05b80ddd328a75e03274f4deaca09dbd709b6c122b06cd49fd349748a7384cffce9e3297ce45d.Audiocodes
loading license key: Audiocodes

insmod: add-symbol-file build/debug/avalanche_cpmac.o 0xc009c060 -s .data 0xc00ad2b0 -s .bss 0xc00adc60
loading license key: Audiocodes
Press ESC to enter BOOT MENU mode.
ata_pages_init,52 --
[mt_rg_conf_compat:3778] saved_version=4.10.4.5.2, rg_version=4.10.4.5.2, saved_ext_version=3.0.1_p041_build_19, external_version=3.0.1_p041_build_19 .
[mt_rg_conf_compat:3779] UPDATING SERIAL NUMBER .
ACL_GetSerialNumberFromMAC: mac_str = 00:90:8F:2F:9B:92
ACL_GetSerialNumberFromMAC: iserial_num=3120018 .
[mt_rg_conf_compat:3793] updating remote access path
MAPS:
00400000-0090d000 r-xp 00000000 1f:02 158032     /mnt/cramfs/bin/openrg
10000000-1009c000 rw-p 0050d000 1f:02 158032     /mnt/cramfs/bin/openrg
1009c000-10158000 rwxp 00000000 00:00 0
7ffInitializing env_params module
f7000-7fff8000 rwxp 00000000 00:00 0

insmod: add-symbol-file build/debug/ac49x_env_mod.o 0xc00c0060 -s .data 0xc00c0300 -s .bss 0xc00c0360

insmod: add-symbol-file build/debug/ac49x_led_mod.o 0xc00c2060 -s .data 0xc00c3f30 -s .bss 0xc00c3f70

insmod: add-symbol-file build/debug/log_chardev.o 0xc00c6060 -s .data 0xc00c6960 -s .bss 0xc00c69e0

insmod: add-symbol-file build/debug/igmp_proxy_mod.o 0xc00c8060 -s .main_flow 0xc00cdd64 -s .data 0xc00ce2a0 -s .bss 0xc00ce3a0

insmod: add-symbol-file build/debug/tcp_mss.o 0xc00d0060 -s .data 0xc00d06e0 -s .bss 0xc00d0710

insmod: add-symbIPV4 device driver registered
ol-file build/debug/rg_ipv4.o 0xc00d2060 -s .data 0xc00d2290 -s .bss 0xc00d22e0

insmod: add-symbol-file build/debug/pppoe_relay.o 0xc00d4060 -s .data 0xc00d6db0 -s .bss 0xc00d6db0

insmod: add-symbol-file build/debug/rg_pppoe_relay.o 0xc00eb060 -s .data 0xc00eb7e0 -s .bss 0xc00eb870

insmod: add-symbol-file build/debug/ac49x_dsp_mod.o 0xc00ed060 -s .data 0xc0106450 -s .bss 0xc0185bf0
dsp memory at block 14c00000
starting vlynq_vdsp configuration...
Failed to initialize the vlynq @ 0xa8611c00.
error msg = Error in configuring clocks..
failed on AcVlynq_Init_SoC_VlynqHi__to__AC498_VlynqLow possibly no vlynq
putting vlynq back in reset...

insmod: add-symbol-file build/deDSP-TELE: SPCR1 reg value = 0x10800FF .
bug/ac49x_mpi_moDSP-TELE: Telephony Interface driver installed.
d.o 0xc018b060 -s .data 0xc018bc10 -s .bss 0xc018bc60

insmod: add-symbol-file build/debug/silabs_daa_3050_drv_mod.o 0xDAA Driver init_module !!!
c018d060 -s .data 0xc0190190 -s .bss 0xc0190280

insmod: add-symbol-file build/debug/silabs_fxs_3215_drv_mod.o 0xc0192060 -s .data 0xc0197fe0 -s .bss 0xc0198e70

insmod: add-symbol-file build/debug/rtp.o 0xc019b060 -s .data 0xc019e1c0 -s .bss 0xc019e1f0

insmod: add-symbol-file build/debug/phone_mod.o 0xc01a0060 -s .data 0xc01a1090 -s .bss 0xc01a10e0

insmod: add-symbol-file build/debug/ac494_mod.o 0xc01a3060 -s .data 0xc01ab120 -s .bss 0xc02277d0

****** AC49x init_module *****

insmod: add-symbol-file build/debug/qos_ingress.o 0xc0238060 -s .main_flow 0xc02Freeing modfs memory: 2036k freed
38980 -s .data 0xc0238b50 -s .bss 0xc0238bf0
factory dft = 0
mt_rg_conf_bad_reboot_verify(). Reboot reason 1
 1 consecutive reboots
getMp20XProdName:116 Need to check g_rg_prod_name (MP20X)
[init:234]
starting voip task watchdog
[AutoDialer_Start: 821]->
[init: 781]->
[init_connection_list: 273]-> num of object list(3)
[init_connection_list: 292]allocating (2) object
[init_connection_obj: 248]dialer type is DHCP
[init_connection_obj: 249]dialer retries(4)
[init_connection_obj: 250]dialer retries timeout(5)
[init_connection_list: 292]allocating (1) object
[init_connection_obj: 220]dialer UserName(fprod-13@INetvision)
[init_connection_obj: 221]dialer Password(827456)
[init_connection_obj: 248]dialer type is PPPOE
[init_connection_obj: 249]dialer retries(4)
[init_connection_obj: 250]dialer retries timeout(5)
[init_connection_list: 292]allocating (0) object
[init_connection_obj: 220]dialer UserName(fprod-12)
[init_connection_obj: 221]dialer Password(676261)
[init_connection_obj: 236]dialer ServerIp(cable.netvision.net.il)
[init_connection_obj: 248]dialer type is L2TP
[init_connection_obj: 249]dialer retries(4)
[init_connection_obj: 250]dialer retries timeout(5)
[init_connection_list: 309]<-
[test_status_init: 325]mgt_int_conn_diag_t(0x10194110) size(464)
[test_status_init: 337]mgt_icd_status_t(0x10193190) size(20) test_id(1)
[test_status_init: 337]mgt_icd_status_t(0x10193428) size(20) test_id(2)
[test_status_init: 337]mgt_icd_status_t(0x10193440) size(20) test_id(4)
[test_status_init: 337]mgt_icd_status_t(0x10193458) size(20) test_id(16)
[init: 812]JL need to add led init here!!!
[init: 813]<-
[AutoDialer_Start: 833]<-
VoIP task started
[test_phys_conn_cb: 757]->
[test_start: 480]->
[test_start: 482]<-
[get_test_results: 494]->
[get_test_results: 498]physical link test is in progress
[get_test_results: 502]
tests results are: status(ICD_FAILURE) spesific(ICD_PHYS_LINK_ETH)
[get_test_results: 504]<-
[test_phys_conn_cb: 761]Failed detecting physical connection
[controller: 516]->
[controller: 528]Auto dialer detection failed. restarting process. 0 retries left
[reset_dialer_list_parameters: 427]->
[reset_dialer_list_parameters: 435]<-
[controller: 532]<-
[test_phys_conn_cb: 769]<-
[test_phys_conn_cb: 757]->
[test_start: 480]->
[test_start: 482]<-
[get_test_results: 494]->
[get_test_results: 498]physical link test is in progress
[get_test_results: 502]
tests results are: status(ICD_FAILURE) spesific(ICD_PHYS_LINK_ETH)
[get_test_results: 504]<-
[test_phys_conn_cb: 761]Failed detecting physical connection
[controller: 516]->
[controller: 536]Auto dialer detection process failed
[controller: 538]JL need to change led status to slow red flash and set dialer to dhcp

Auto Dialer Detect Statistics Summery
-------------------------------------

dialer name: L2TP
status: not tested
time spent on dialer connection: 0 sec

dialer name: PPPOE
status: not tested
time spent on dialer connection: 0 sec

dialer name: DHCP
status: not tested
time spent on dialer connection: 0 sec

total running time: 0 seconds
[uninit: 402]->
[uninit_connection_list: 376]->
[uninit_connection_list: 391]<-
[uninit: 415]<-
[controller: 546]<-
[test_phys_conn_cb: 769]<-
[voip.c:1507] checking SLICIOC_R_FXS_READ_INIT_DONE for dev /dev/ac494_slic1
[voip.c:1526] init_done = 0.
[voip.c:1535] setup a new timer .
Username: Failed detecting physical connection

Failed detecting physical connection

Auto dialer detection process failed

[mt_voip_ipc 870] voip is ready for ipc
voip: new configuration parameters loaded
[voip_conf_get_number_of_dsp_devices:3342] iNumO----------i = DSP_REGIONAL_SETTINGS_CALL_PROGRESS_CALL_WAITING_TONE ----------
fDspDevices=1, iac494_configure_dsp: about to configure time slots, iNumOfAnalogChannels=2.
First=1.
ac494_configure_dsp: THE END !!!!
TimerInit: pTimerObj->ticksPerSecond=0x64 .
CountryName=USA
ACL_Si321x_GetCountryIndex: Found a match (country=USA, index=0) .
Si321x_ZsynthSetup
Si3215-16 Rev C
CountryName=USA
ACL_Si321x_GetCountryIndex: Found a match (country=USA, index=0) .
Si321x_ZsynthSetup
Si3215-16 Rev C
[phone_api.c:60] slic_status_polling ...
[phone_api.c 74] polling on slic status(1)

[phone_api.c 106] passed slic_status_polling
[voip.c:1507] checking SLICIOC_R_FXS_READ_INIT_DSilabs drv: SLICIOC_W_FXS_WRITE_CALIBRATION_RESULTS_PRECONFIG (channel=0, scFXSCalibrationWriteCounter=1) .
ONE for dev /dev Reg98=16, Reg99=17, Reg100=17, Reg101=17, Reg102=5, Reg103=54, Reg105=216, Reg107=8
/ac494_slic1
[vslic_ioctl:cmd BLOCKED=4004eea1 runing=8004ee90
oip.c:1526] initerror reading sys_ioctl
_done = 1.
[voip.c:1543] FXS Calibration is notSilabs drv: SLICIOC_W_FXS_WRITE_CALIBRATION_RESULTS_PRECONFIG (channel=1, scFXSCalibrationWriteCounter=2) .
 required, use pslic_ioctl:cmd BLOCKED=4004eea1 runing=8004ee90
reconfiured paraerror reading sys_ioctl
ms.
 [voip.c 1 Reg98=15, Reg99=15, Reg100=17, Reg101=14, Reg102=7, Reg103=87, Reg105=43, Reg107=7
393] mt_voip_slic_calibration_done_parameters_poll: Calibration Results:
 Reg98=16, Reg99=17, Reg100=17, Reg101=17, Reg102=5, Reg103=54, Reg105=216, Reg107=8
 [voip.c 1393] mt_voip_slic_calibration_done_parameters_poll: Calibration Results:
 Reg98=15, Reg99=15, Reg100=17, Reg101=14, Reg102=7, Reg103=87, Reg105=43, Reg107=7
RV conf using localAddress 10.100.102.1
RV conf using userDomain 10.100.102.1
calling rvIppSipStackInitialize
rvIppSipStackInitialize returned OK
**** mt_voip_ext_conf_poll: num_of_fxos_lines=2, num_of_fxo_lines=0 

 acl_rg_conf_voip_ext_commit
memory_watchdog: interval=10000, memory_margin=3500 .
rmt_cfg_start>>
automatic dialer detection process done!
automatic dialer detection process failed!

The boot loader is PSP Boot, of course.

Add-on icons vanished after Firefox upgrade

Don’t get me wrong: I didn’t upgrade Firefox. It upgraded itself suddenly into Firefox 68.o Quantum. How did I notice? Not because it became any better, but because suddenly things stopped working. And I got logged out from several sites.

For example, my Adblocker and other add-on suddenly vanished. Checking the add-on settings, the relevant add-ons were installed and enabled, so what?

Well, it turns out that I had to manually add the icons to the toolbar. As simple as that: Right-click the toolbar where the icons usually were, select Customize… and drag the icons where they should be.

Why Firefox upgraded itself is beyond me. Whenever possible, I always opt this out (doesn’t look like Firefox has such option). So if any of the Firefox maintainers ever read this: If it was a security issue, patch it. Don’t push a whole upgrade I didn’t ask for. A want my computer as stable and repeatable as you want your own car. In case you care about it, that is.

Intel FPGA’s Stratix 10: My impressions and notes

Introduction

These are a few random things I wrote down as I worked with the Stratix 10 Development Kit, with focus on its PCIe interface. Quite obviously, it’s mostly about things I found noteworthy about this specific FPGA and its board, compared with previous hardware I’ve encountered.

Generally speaking, Stratix 10 is not for the faint-hearted: It has quite a few special issues that require attention when designing with it (some detailed below), and it’s clearly designed with the assumption that if you’re working with this king-sized beast, you’re most likely part of some high-end project, being far from a novice in the FPGA field.

Some National Geographic

Even though I discuss the development kit further below, I’ll start with a couple of images of the board’s front and back. This 200W piece of logic has a liquid cooler and an exceptionally noisy fan — none of which are shown in Intel’s official images I’ve seen. In other words, it’s not as innocent as it may appear from the official pics.

There are no earplugs in the kit itself, so it’s recommended to buy something of that sort along with it. One could only wish for a temperature controlled fan. I mean, measuring the temperature of the liquid would probably have done the job. Some silence when the device isn’t working hard.

So here’s what the board looks like out of the box (in particular DIP switches in the default positions). Click images to enlarge.

Front side of Stratix 10 Development Kit

Front side of Stratix 10 Development Kit

Back side of Stratix 10 Development Kit

Back side of Stratix 10 Development Kit

 

“Hyperflex”

The logic on the Stratix 10 FPGAs has been given this rather promising name, implying that there’s something groundbreaking about it. However synthesizing a real-life design for Stratix 10, I experienced no advantage over Cyclone 10: All of the hyper-something phases got their moment of glory during the project implementation (Quartus Pro 19.2), but frankly speaking, when the design got the slightest heavy (5% of the FPGA resources, but still a 256-bit wide bus everywhere on a 250 MHz clock), timing failed exactly as it would on a Cyclone 10.

Comparing with Xilinx, it feels a bit like Kintex-7 (mainline speed grade -2), in terms of the logic’s timing performance. Maybe if the logic design is tuned to fit the architecture, there’s a difference.

Assuming that this Hyperflex thing is more than just a marketing buzz, I imagine that the features of this architecture are taken advantage of in Intel’s own IP cores for certain tasks (with extensive pipelining?). Just don’t expect anything hyper to happen when implementing your own plain design.

PCIe, Transceivers and Tiles

It’s quite common to use the term “tiles” in the FPGA industry to describe sections on the silicon die that belong to a certain functionality. However the PCIe + transceiver tiles on a Stratix 10 are separate silicon dies on the package substrate, connected to the main logic fabric (“HyperFlex”) through Intel’s Embedded Multi-die Interconnect Bridge (EMIB) interface. Not that it really matters, but anyhow.

H, L and E tiles provide Gigabit transceivers. H and L tiles come with exactly one PCIe hard IP each, E-tiles with 100G Ethernet. There might be one or more of these tiles on a Stratix 10 device. It seems like the L tile will vanish with time, as it has weaker performance in almost all parameters.

All tiles have 24 Gigabit transceivers. Those not used by the hard IP are vacant for general purpose, even though some might become unusable, subject to certain rules (given in the relevant user guides).

And here comes the hard nut: PCIe has a minimal data interface of 256 bits with the application logic. The other possibility is 512 bits. This can be a significant burden when porting a design from earlier FPGA families, in particular if they were based upon a narrower data interface.

Xillybus supports the Stratix 10 device family, however.

PCIe unsupported request error

Quite interestingly, there were correctable (and hence practically harmless) errors on the PCIe link consistently when booting a PC with the official development kit, with a production grade (i.e. not ES) H-tile FPGA. This is what plain lspci -vv gave me, even before the application logic got a chance to do anything:

01:00.0 Unassigned class [ff00]: Altera Corporation Device ebeb (rev 01)
        Subsystem: Altera Corporation Device ebeb
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at d0100000 (64-bit, prefetchable) [size=256]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit-
                Address: 00000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 5GT/s, Width x16, ASPM not supported, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

As highlighted above, Unsupported Request correctable errors took place on the link. Even though this is harmless, it’s nevertheless nothing that should happen on a properly working PCIe link.

Note that I ran the PCIe link on Gen2 only, even though it supports Gen3. Not that it should matter.

Reset release IP

According to Intel’s Configuration Guide for Stratix 10 for Quartus Design Suite 19.2, one can’t rely on the device’s consistent wakeup, but the nINIT_DONE signal must be used to reset all logic:

“The entire device does not enter user mode simultaneously. Intel requires you to include the Intel Stratix 10 Reset Release IP on page 22 to hold your application logic in the reset state until the entire FPGA fabric is in user mode. Failure to include this IP in your design may result in intermittent application logic failures.”

Note nINIT_DONE is asserted (low) when it’s fine to run the logic, so it’s effective an active HIGH reset. It’s so easy to get confused, as the “n” prefix triggers the “active low reset” part of an FPGA designer’s brain.

Failing to have the Reset Release IP included in the project results the following critical warning during synthesis (Quartus Pro 19.2):

Critical Warning (20615): Use the Reset Release IP in Intel Stratix 10 designs to ensure a successful configuration. For more information about the Reset Release IP, refer to the Intel Stratix 10 Configuration User Guide.

The IP just exposes the nINIT_DONE signal as an output and has no parameters. It boils down to the following:

wire ninit_done;
altera_s10_user_rst_clkgate init_reset(.ninit_done(ninit_done));

One could instantiate this directly, but it’s not clear if this is Quartus forward compatible, and it won’t silence the critical warning.

However Quartus Pro 18.0 doesn’t issue any warning if the Reset Release IP is missing, and neither is this issue mentioned in the related configuration guide. Actually, the required IP isn’t available on Quartus Pro 18.0. This issue obviously evolved with time.

Variable core voltage (SmartVID)

Another ramp-up in the usage complexity is the core voltage supply. The good old practice is to set the power supply to whatever voltage the datasheet requires, but no, Stratix 10 FPGAs need to control the power supply, in order to achieve the exact voltage that is required for each specific device. So there’s now a Power Management User Guide to tackle this issue.

This has a reason: As the transistors get smaller, so does the tolerance of the process get a larger impact. To compensate for these tolerances, and not take a hit on the timing performance, each device has its own ideal core voltage. So if you’ve gone as far as using a Stratix 10 FPGA, what’s connecting a few I2C wires to the power supply and let it pick its favorite voltage?

The impact on the FPGA design is the need to inform the tools which pins to use for this purpose, what I2C address to use, which power supply to expect on the other end, and other parameters. This takes the form of a few extra lines, as shown below for the Stratix 10 Development Kit:

set_global_assignment -name USE_PWRMGT_SCL SDM_IO14
set_global_assignment -name USE_PWRMGT_SDA SDM_IO11
set_global_assignment -name VID_OPERATION_MODE "PMBUS MASTER"
set_global_assignment -name PWRMGT_BUS_SPEED_MODE "400 KHZ"
set_global_assignment -name PWRMGT_SLAVE_DEVICE_TYPE LTM4677
set_global_assignment -name PWRMGT_SLAVE_DEVICE0_ADDRESS 4F
set_global_assignment -name PWRMGT_SLAVE_DEVICE1_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE2_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE3_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE4_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE5_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE6_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE7_ADDRESS 00
set_global_assignment -name PWRMGT_PAGE_COMMAND_ENABLE ON
set_global_assignment -name PWRMGT_VOLTAGE_OUTPUT_FORMAT "AUTO DISCOVERY"
set_global_assignment -name PWRMGT_TRANSLATED_VOLTAGE_VALUE_UNIT VOLTS

It’s among the things that are easy when they work, but when designing your own board and something goes wrong with the I2C bus, well, well.

“Self service licensing”

The Stratix 10 Development Kit includes a one-year license for Quartus Pro, which is activated on Intel’s website. It’s recommended to start this process as soon as possible, as it has a potential of getting tangled and time consuming. In particular, be sure to know which email address was reported to Intel along with the purchase of the Kit, and that you have a fully verified account for that email address on Intel’s website.

That’s because the self-service licensing procedure is possible only from the Intel account that is registered with a specific email address. This email address is the one that the distributor reported when forwarding the order for the development kit to Intel. In my case, they used an address they had on record from a previous purchase I made from the same distributor, and it didn’t even cross my mind to try it.

Be sure to fill in the detailed registration form and to confirm the email address. Access to the licensing area is denied otherwise. It continues to be denied for a few days after filling in the details. Probably a matter of validation by a human.

The serial number that needs to be fed in (or does it? see below) is the one that appears virtually everywhere (on the PCB itself, on the package, on the outer box with which the package arrived), and has the form of e.g. 10SHTPCIe0001234. However the instructions said it should be “printed on the side of the development kit box below the bottom bar code”. Well, there is nothing printed under the bottom bar code. It’s not so difficult to find it, as it says “serial number”, but when the registration fails, this misleading direction adds a level of confusion.

Since the serial number is so out in the open, it’s quite clear why another form of authentication is needed. Too bad that the email issue wasn’t mentioned in the instructions.

In my case, there was no need to feed any serial number. Once the Intel account was validated (a few days after filling in the registration details), the license simply appeared on the self-service licensing page. As I contacted Intel’s licensing support twice throughout the process, it’s possible that someone at Intel’s support took care of pairing the serial number  with my account.

Development kit’s power supplies

I put this section last, because it’s the one that is quite pointless reading, frankly speaking. The bottom line is simple, exactly like the user guide says: If you use the board stand-alone, use the power supply that came along with it. If the board goes into the PCIe slot, connect both J26 and J27 to the computer’s ATX power supply, or the board will not power up.

J27 is a plain PCIe power connector (6 pins), and J26 is an 8-pin power connector. On my plain ATX power supply there was a PCIe power connector with a pair of extra pins attached with a cable tie (8 pins total). It fit in nicely into J26, it worked, no smoke came out, so I guess that’s the way it should be done. See pictures below, click to enlarge.

ATX power supply connected to Stratix 10 Development Kit, front side

ATX power supply connected to Stratix 10 Development Kit, front side

ATX power supply connected to Stratix 10 Development Kit, back side

ATX power supply connected to Stratix 10 Development Kit, back side

Now to the part you can safely skip:

As the board is rated at 240 W and may draw up to 20A from its internal +12V power supply, it might be interesting understand how the power load is distributed among the different sources. However the gory details have little practical importance, as the board won’t power up when plugged in as a PCIe card unless power is applied both to J26 and J27 (the power-up is sequencer set up this way, I guess). So this is just a little bit of theory.

There are three power groups, each having a separate 12V power rail: 12V_GROUP1, 12V_GROUP2 and 12V_GROUP3.

12V_GROUP2 will feed 12V_GROUP1 and 12V_GROUP3 with current if their voltage is lower than its own, by virtue of an emulated ideal diode. It’s as if there was two ideal diodes connected with their anodes on 12V_GROUP2 and one diode’s cathode on 12V_GROUP1, and cathode on 12V_GROUP3.

These voltage rails are in turn fed by external connectors, through emulated ideal diodes as follows:

  • J26 (8-pin aux voltage) feeds 12V_GROUP1
  • J27 (6-pin PCIe / power brick) feeds 12V_GROUP2
  • The PCIe slot’s 12V supply feeds 12V_GROUP3

The PCIe slot’s 3.3V supply is not used by the board.

This arrangement makes sense: If the board is used standalone, the brick power supply is connected to J27, and feeds all three groups. When used in a PCIe slot, the slot itself can only power 12V_GROUP3, so by itself, the board can’t power up. Theoretically speaking, J27 needs to be connected to the computer’s power supply through a PCIe power connector, at the very least. For the higher power applications, J26 should be connected as well to the power supply, to allow for the higher current flow. In practice, J27 alone won’t power the board up, probably as a safety measure.

The FPGA’s core voltage is S10_VCC, which is generated from 12V_GROUP1 — this is the heavy lifting, and it’s not surprising that it’s connected to J26, which is intended for the higher currents.

The ideal diode emulation is done with LTC4357 devices, which measure the voltage between the emulated diode’s anode and cathode. If this voltage is slightly positive, the device opens a external power FET by applying voltage to its gate. This FET’s drain and source pins are connected to the emulated diode’s anode and cathode pins, so all in all, when there’s a positive voltage across it, current flows. This reduces the voltage drop considerably, allowing efficient power supply OR-ing, as done extensively on this development kit.

The board’s user guide advises against connecting the brick power supply to J27 when the board is in a PCIe slot, but also mentions the ideal diode mechanism (once again, it won’t power up at all this way). This is understandable, as doing so will cause current to be drawn from the PCIe slot’s 12V supply when its voltage is higher that the one supplied by J27, even momentarily. With the voltage turbulence that is typical to switching power supplies, the currents may end up swinging quite a lot in an unfortunate combination of power supplies.

So even though it’s often more comfortable to control the power of the board separately from the hosting computer’s power, or to connect J27 only if the board is expected to draw less than 75W, both possibilities are eliminated. Both the noisy fan and the board’s refusal to power up unless fed properly prepare the board for the worst case power consumption scenario.

KiCad notes: The one-timer PCB maker’s hints for the next time

Intro

These are my notes to self for the next time I’ll have a PCB design to make. This is definitely not my expertise.

My original intention was to start this post with some kind of apologize, saying I only needed to make a simple board, so KiCad was good enough for this purpose. However it seems like KiCad is about to become the preferred tool many PCB designers quite soon, if it’s not there already. It’s surely handy and relatively easy to learn (at least for me, an experienced Electrical Engineer with no previous hands-on in board design). Does it match up with the expensive tools out there for a heavy and complicated design? I don’t know.

Everything in KiCad is text based in a simple manner. Important for version control, and using it to monitor my changes. Also very handy for fiddling with the files to get the things right on spot. In particular useful when creating footprints.

All said here relates to KiCad 5.1.4 under Linux Mint 19.

Getting started

  • The KiCad version that went along with Mint 19 was heavily outdated. Compare what packages are available directly from KiCad.
  • When upgrading from 4.x to 5.x, be sure to uninstall the previous version and purge it (apt purge) before installing the new one. Also, delete .config/kicad/ as it contains the paths to libraries for symbols and footprints.
  • The ‘Footprint library path “…” does not exist’ error may be a result of not cleaning up an old installation as mentioned in the previous item.
  • The official documentation, in particular the Getting Started with KiCad guide, is definitely the right place to start and refresh memory.

EESchema

  • Before starting to work: Verify that the page size is correct and that the grid is 50 mils or so, no lower: View > Grid Settings… to ensure this.
  • Basic components: In the “Device” library.
  • Aiming the cursor on something accurately and right-clicking is usually the start for modifying virtually anything.
  • Better still: Note the hotkey and use it while hovering over the relevant place.
  • “Drag” keeps connections. “Move” doesn’t. Use “Drag”.
  • If wires don’t snap to the ports, and components are placed regardless to the grid, it’s most likely that the grid is too dense.
  • Useful hotkeys: Hover-Delete (well, delete), Hover-G (drag, think Blender) and Hover-C (duplicate).
  • “Value”: Each component has a value assigned to it. The value appears on the assembly layer (F.Fab or B.Fab) of the PCB. In other words, the person doing the assembly of the board will see this value in the component’s position, and select what to put there accordingly. For resistors and capacitors, it’s obviously the value of the component. For other components, a useful hint on the nature of the device should be there.
  • “Reference”: The symbol’s unique identifier in the schematics and netlist. U1, J15, C225 etc. By default, they’re C?, R? etc. until Annotate Schematic… is done.
  • Net names: Designated with “Place Net Label” or by pressing “l”. Makes things easier during layout, and differential pairs must be with “+” and “-” suffixes (or _p/_n), or PCBNew won’t play ball later on.
  • Footprints are assigned on each component’s Properties (and are not associated with symbols, unlike other tools), or more conveniently, with Tools > Assign Footprints. The association affects the schematic (.sch) file, appearing as a string property consisting of the library name and the footprint’s name, separated by a colon. Don’t associate to library footprints, but copies of them, defined explicitly on a local library. Or upgrading KiCad may change the design. See below on footprints.
  • Plain holes are “virtual components”. Add a MountingHole symbol for each, and assign a matching footprint (hole only).

Symbol Editor

  • Read the Symbol Guidelines in KiCad Library Convention document (KLC), after the daunting naming convention part (start at S3, maybe). It’s how to get it right.
  • Check the grid. 50-100 mils, or the wires won’t snap to ports when the symbol is used (official guideline is 100 mils).
  • Symbol Editor: When drawing a rectangle, click at the corner points, don’t hold the mouse button.
  • It doesn’t hurt setting the “Description” property for each symbol — it appears on the library’s list as well as the output netlist.
  • If a pin has multiple pads assigned to it, KiCad expects them to have track wires on the PCB connecting between them all.

Understanding footprints

Footprints are a set of graphical shapes in different layers that serve several purposes, all of which much be addressed when setting up a custom footprint:

  • Putting the pads in place: Front copper layer (F. Cu, and also B. Cu with thru pads) and solder mask (F. Mask). Actually, the solder mask layer is the hole in the solder mask. Even though the shape of these is the same, the solder mask is widened slightly when Gerber files are created.
  • Solder paste: When used, where the stencil holes should allow solder paste to be applied (F. Paste). As with solder mask, the shape is usually the same as the pads, but the datasheet may require a different pattern.
  • Holes (drilling): These are a side effect of thru-hole pads, and appear in the footprint layers as is there was copper there.
  • Silk Screen: Descriptive information on the board, in particular the component reference (F. SilkS)
  • Assembly: Mark the place where the device actually is (F. Fab). The placement’s origin should also be the footprint’s (x,y) origin for automatic placement. The reference designator (“%R”) is written inside the component’s boundaries, and its “value” (what it is) typically underneath.
  • Clearance: Ensure that nothing else is placed in the space that the device requires. This no-place space is called “courtyard” (F. CrtYd). The idea is simple: If, during layout (with PCBnew), the courtyards of two components overlap, it’s a DRC error at the very least, and possible also a refusal to generate Gerber files. So the courtyard should be the area that the component must occupy exclusively.

The layer assignment assumes that the component is placed on the front. When placing on the back side, the Layout Editor flips the layer assignment as necessary.

Holes is a bit tricky, because they are an implicit result of pads with holes. In particular, NPTH is Non-Plated Thru Holes, and appear in the footprints as pads of this type.

Footprints

There’s another post of mine, which discusses pads and footprint manipulation.

  • Rule number one: Copy or imitate an existing library footprint rather than reinventing the wheel. Preferably imitate one that is likely to be heavily used.
  • Read the Footprint Guidelines in KiCad Library Convention document (KLC). Skip the naming convention, and start at F4 (or so). This might be the best source on the planet for setting up a footprint. Odds are that these rules are tuned based upon real-life experience of a lot of people.
  • Never ever use a footprint directly from the libraries to a component in the design. That makes the outcome dependent on the KiCad version running.
  • Rather, create a project-dedicated footprint library (a directory with a .pretty suffix) in the Footprint editor, and put it in the Project’s library table. Then open the desired footprint in Footprint editor, and save it into the project’s library.
  • Note that the 3D model still points at a KiCad library even after this, but that’s less important.
  • Monitor the changes with git. It’s quite straightforward and allows a safe double-check.
  • Don’t hesitate editing the footprint file manually, in particular for nailing exact positions. Exit the Footprint Editor and re-enter it, and then load the footprint. Using “Revert” for updating changes won’t work.
  • To verify that a footprint is OK:
    • At the very least view each layer separately, and make sure it contains what it should. Drill holes are covered by absence of material in the copper layers.
    • Verify that SMT pads cover F.Cu, F.Mask and F.Paste (check each separately!). For thru pins, also on the back.
    • Read through the text file (with _mod suffix) and look for unexpected stuff (one pad having different dimensions for no reason etc.)
  • Pads may not have a pad number. This is useful for e.g. stencil pattern not matching the shape of the pad.
  • If multiple pads have the same pin number assigned to them, track wires must be connected between them during the layout.
  • Pads and other elements may very well not be on the footprint nor layout grid. I find myself calculating their X/Y position rather than placing them, so any position goes.
  • For adding anything other than a pad, the drawing or text falls on the layer marked with a small arrow in the list of layers.
  • Text substitutions: %R is replaced with the reference, %V with the value.
  • Even though the name of the footprint is shown as the “value” in the Footprint Editor, it will be the component’s “value” field in Pcbnew. So it’s useful information during assembly of the board.
  • Right-click on a pad, select “Array” to produce several identical pads with equal spacing. In extreme cases, write a simple script to manipulate the footprint file directly (exotic positions or pad naming).
  • Use the measurement tool for assurance. It snaps to the grid.
  • Any line thickness is implemented as a filled circle drawing along the line. This makes rounded corners.
  • Pads can have arbitrary shapes. See this post.

PCBnew (layout)

  • When starting a new project, remove the *.kicad_pcb file as well as .config/kicad directory, and start from scratch. KiCad’s defaults are usually good, so if a detail is overlooked, better have the default than some previous setting.
  • File > Board Setup… and set the board’s thickness, trace parameters etc. It’s also a good idea to define the net classes (in particular high frequency nets).
  • PCBnew depends on the current netlist as well as the footprint libraries. They are updated only on explicit request, so each time the schematics or footprints in use are updated, the changes must be synchronized by user request. As for the schematics, there an “Update PCB from Schematics” button. Or export the netlist from EESchema in default format and load it into PCBnew. It’s fine to do this iteratively (and even finer when covered by version control to cover up for mishaps).
  • Grid: Pads may be off the grid, and there’s no problem selecting them as a start and end point for a route. The track itself will generally go on the grid, except for leaving it in order to reach a pad in a sane manner (usually with a 45-degree detour, not on a grid point). The grid pitch should hence be the approximately most used track width + the track clearance if a dense design is desired. Possibly make it in line with crucial footprints’ pitch, to keep straight traces, aligned with the grid.
  • Ratsnest: Straight lines showing connections between pads. View all at once with View > Show Ratsnest (default on) or turn off, and view individual lines with “Display local ratsnets” icon on the toolbar to the right.
  • The origin’s position has no significance, expect that it places the grid. However the absolute position values appear in the Gerber files, but this can be fixed by adding an auxiliary axis (more on this below).
  • There are no 90 degree turns on tracks (except for power lines), only 45 degrees (and possibly smaller angles when aligning to the grid). Feels a bit awkward in the beginning but sharp turns are evil.
  • There is no dragging of components (footprints), only moving them, which results in disconnection of all tracks going to it.
  • While drawing tracks, press “v” for a via, press “/” to change the posture (the position of the 45 degree detour, if present).
  • Always start and end a track by clicking at the center of a pad (the “anchor”), in particular if it’s off-grid.
  • Ground / power planes: Create a fill zone, which is a polygon which is designated to a net. After setting up the zone, select it, right-click and pick Zones > Fill. This may not work unless there is already a track with the desired net going through the area. The track may remain, as it’s harmless. It’s probably the via, which connects the area to the net that does the trick. Once set up, a via is enough to connect to the plane.
  • Repeat the zone fill after each change that may affect it.
  • If the zone fill gets too close to a specific footprint’s pads, open the “Local Clearance and Settings” tab on the footprint’s Properties, and set a larger (actually, non-zero) clearance.
  • In the end, tracks are a sequence of segments with a start and end position, width, layer and net designation in the *.kicad_pcb file (with “segment” keyword), vias are elements with the “via” keyword etc. Everything is absolutely positioned. Therefore moving a component (actually, its footprint) disconnects everything. All in all, PCBnew is just a smart drawing program for geometric shapes.
  • Differential pairs: Route > Differential Pair. Then Route > Tune Differential Pair Skew / Phase, in order to check the skew and possibly tune it (or possibly move a component if that helps).
  • Setting the parameters for track length and skew tuning: After selecting the track to tune, instead of moving the mouse to get those humps, right click and select Length Tuning Settings (or CTRL-L). The amplitude and length of these humps can be set, as well as the target length (instead of the default 100 mm).
  • There’s a measurement tool (“Add dimension”) which is handy to verify certain distances, in particular the dimension of the board. The tool shown an indication when it’s on a line end, so one can be sure the accurate measurement is made. Best done on a separate drawing layer.
  • Optionally, add an auxiliary axis origin to the layout, placing the (0,0) point of Gerber and drill files. This changes the aux_axis_origin parameter of the PCB design from (0 0) to whatever it was set to, and the coordinates in the Gerber files will have an offset relative to the layout files. Useful if the entire layout was drawn with some meaningless origin position. Then check “Use auxiliary axis as origin” when generating Gerbers and drill files.

3D viewer

I didn’t understand the point of a 3D view at first, but it’s surprisingly helpful, in particular during PCB layout — even though it’s also handy with the Footprint Editor. There’s something reassuring seeing the silk screen as white text on a green solder mask, and the drill holes as see-throughs. Click the mouse’s middle button to move the scenery.

Launch with View > 3D viewer (or Alt-3).

Generate Gerber files

You have reviewed the DRCs in EESchema and PCBnew, right…?

So to the final check before taping out. Export the files in PCBnew with File > Plot… and as follows (for a two-layer design):

Screenshot of dialog box for producing Gerber Files in KiCad(click image to enlarge)

and then generate a drill file (click “Generate Drill File”, of course).

Screenshot of dialog box for producing drill files in KiCad(click image to enlarge)

All the settings above are KiCad’s defaults, by the way. Changes in these settings are stored in the *.kicad_pcb file for the next time these windows are opened (and not KiCad-globally).

Note that two drill files are (usually) generated: One for plated holes (vias and pads) and one for non-plated holes (typically mechanical holes for mounting).

A component positioning file can be generated with Fabrication Outputs > Footprint Position (.pos) file, however this file uses the internal footprints and their names. In particular, if the footprint’s origin isn’t where the automatic placement machine thinks it should be, the device may end up in the wrong place.

Viewing Gerber files

KiCad’s Gerber is fine, but it’s recommended to install gerbv as well (plain “apt install gerbv” on Mint 19). It’s part of gEDA, and it allows changing the order of the layers, but even more important, to export the current view to pdf. KiCad’s viewer prints one layer per page, and it’s hardly helpful. Use the export feature, don’t print — the latter generates a microscopic printout.

The main motivation for printing layers is to print the fabrication layer (preferably without values, KiCad has that option) on top of the edge cut and possibly a copper layer. For manual assembly, this gives a good indication of what goes where. To run it, just go

$ gerbv &

For KiCad’s viewer, load both the Gerber files and the drill files into Gerbview, and take a close look. The layers are displayed in the order they were loaded, so load the drill files first. As they are usually loaded with multiple selection of files, that’s probably alphabetical order, putting the bottom layer above the front.

Right-click on the layer list and select “Sort Layers if X2 mode” (should be “in X2 mode”, I guess) for layers ordered in physical order.

The active layer is always visible (drawn last).

Most important: Verify that the drill holes, plated and not, are in the correct places.

The DCodes are the aperture with which each line is drawn. An aperture is like a paintbrush, which is defined in each Gerber file for its own drawings. It’s usually a filled circle or rectangle with specific dimensions. The benefit of having these on the screen is to identify tracks with the same width and pads with the same dimensions, if they have the same DCode and are on the same layer. The same DCode on different layers (and hence Gerber files) has no meaning. For example, D10 and D11 are typically the DCodes of the first apertures defined in Gerber file, and are practically on every layer.

Create assembly sheet

If the design is going to be hand-assembled, better have a simple drawing that tells what goes where. To do this, plot the Gerber files as before, but uncheck “Plot footprint values” (but keep “Plot footprint references” checked, of course). Check F.Fab layer for plotting.

Recall that PCBnew remembers this in the *.kicad_pcb file, so be sure to revert it (with git) afterwards.

Next, open the files with gerbv (not KiCad’s Gerber viewer). Display the F_Fab layer, which should have the component references printed small inside boxes, along with the edge cuts and a copper layer. Change the colors for convenience.

Then File > Export > SVG… and generate a file. Import that file into a Libreoffice Writer document, and export that document as a pdf. Why not directly as pdf? Because the output pdf is broken, and even when I managed to open it, turning it into a pdf with a decent size turned out impossible.

A similar SVG export can be done from PCBNew: Pick File > Export > SVG… and select “Board area only” under “SVG Page Size”. For Pagination, pick “All layers in a single file”. However I haven’t figured out how to get rid of the values, and they clutter everything.

Manufacture

  • Holes: Manufacturers typically make the hole slightly larger than required, as commented on this thread. The difference is typically no more than 0.1mm, so it’s not a big deal.
  • PCB material: FR-4 is de-facto industry standard for rigid boards. 1 oz copper thickness is by far the most common. Pick thicker only for high currents. Board thickness: 0.6/0.8/1.0/1.2/1.6/2.0 mm. Tolerance is typically ±0.1 mm.
  • Drill size: 0.2 – 6.3mm (but don’t go lower than 0.4 mm = 16 mils for basic PCB)
  • Trace width: 6 mil seems to be a common minimal trace width, but Protel used to set 10 mil as the default (the latter seems to carry up to 1A, but check with a calculator). Unless there are current to carry, stick to these figures in general.
  • Minimal trace space: 3 mil seems to be possible, but 10 mils should be chosen unless there’s a reason to push it. Minimal trace to board edge clearance should be 0.3 mm, but standard edge connector footprints tend to take 0.5mm for their pads. So better go for the latter.
  • All in all, it seems like 10 mils for minimal trace width and space, and 0.4mm mils for vias are the sane choice. If the board gets tight, go smaller.
  • If the board has gold fingers, it’s ENIG (Electroless Nickel Immersion Gold) surface finish (sometimes referred to as just Immersion Gold).
  • The assembly is the expensive part. Manufacturing should be at around $5-10 for 5 copies for a small board.
What to submit:
  • Gerber files in RS-274x format for board outline, copper layers, silk screen(s), solder masks and solder paste. NC drill sometimes required in Gerber format instread of Excellon.
  • NC Drill file, in Excellon format
  • For assembly, this page summarizes it well. An assembly sheet as described above is a good idea.

Finding a manufacturer

My board involved small SMT components, so no chance to do the assembly myself. Actually, 10 simple components, on a plain FR4 53 x 22 mm board with two 1 oz layers and a gold finger (hence ENIG finish). The price for a turn-key project is at least $100. Don’t expect less.

And the minimal order quantity is 5. Everywhere.

The place to make a comparison is  PCB shopper. I also suggest look at Manufacturing Reports for some painfully detailed reviews on PCB quality. If you need assembly, select service by assembly costs, as it’s going to be the dominant part.

So these are the estimated prices I got for a small board.

  • Smart Prototyping, assembly at $80.
  • Elecrow, assembly at about $106.
  • PCBWay, assembly at $30 as a Turnkey project, $88 as kitted or partly kitted. Something tells me they’re going to charge a lot for the devices. They do send stuff with regular E-Packet = 16 days.
  • ShenZhen2U, assembly at $50 (pretty much fixed price), allowing plain mail shipping. Headquarters in Hong Kong, but apparently the factory is in Shenzhen (China).

I have no affiliation with any of these companies, and this reflects the situation as of September 2019.

These costs didn’t change much when I changed the parameters of the assembly task. It’s sensible to assume it’s mostly the setup fee.

So I sent the files to PCBWay and ShenZhen2U. My personal impressions:

  • PCBWay: Nice web interface, quick and professional response (an hour or so, during Chinese daytime). The price difference between the turnkey and the kitted was indeed a hint to their tendency to make up for the low assembly cost by adding fees everywhere: The components were 50-100% more expensive than Mouser (in low quantity prices). When I asked about it, I got some BS answer that it’s more expensive for them, taxes etc. PCB manufacturing at $39, which is pretty high. And they add $8 Paypal fee and $20 for shipping and handling with E-packet. And then there’s an “assembly promotion discount” of $30, ending up at $108 next to the “Confirm & Pay” button.
  • ShenZhen2U: Slightly slower response (1 day each time), the website is slightly less useful. The flow was that I ordered the assembly first, then a few emails back and forth (I filled in their BOM form, where I also defined the PCB, and then got an accurate quote) and then the order at the website was updated manually to reflect their quote. They stood behind the $50 assembly cost. Their BOM costs are only slightly higher than Mouser, and they charge $10 for E-packet shipping. PCB manufacturing at $26, which is pretty low. All in all $110, when paid though Paypal (including a Paypal fee of $4.86, which they were clear about and is reasonable).

So there we have it: Almost exactly the same bottom line. One company reached it after fiddling with the prices, and the second just listed them upfront. This is why I went for ShenZhen2U, even though its review on Manufacturing Reviews complained a lot about dirt and bad silkscreen. The parallel review for PCBWay complained about traces being too thin, which is much more serious IMHO.

To be continued as the events unfold…

What to track with git

For a project named projname, this is the list of files I found sufficient to track:

.gitignore
fp-lib-table
projname.dcm
projname.kicad_pcb
projname.lib
projname.pretty/*.kicad_mod
projname.pro
projname.sch
sym-lib-table

as for .gitignore itself, this is my somewhat bloated file, which eventually did the job.

*-save.pro
*-cache.lib
*-rescue.lib
*-save.kicad_pcb
_autosave-*.*
*.kicad_pcb-bak
*.000
*.net
*.net~
*.bak
*.sch-bak
*.bck
*.gbr
*.drl
*.pos
*.wrl
*.csv
*.rpt
*.step
*.stp
*.svg
*.ps
*.pdf
*.dxf
*.plt
*.cad
fp-info-cache
*~

This ignores files produced by KiCad as well as backups. Plus a lot of garlic against files that were never seen.

PCB layout: Selecting the track width and spacing for a differential signal

Introduction

Even though this is definitely not my expertise, I made a simple adapter board carrying USB 3.0 signals.

The name of the game with such boards is to make a minimal impact on the signals, which means, among others, selecting the attributes of these two pairs of differential lines (SSRX and SSTX) in a way that minimizes the damage their very existence causes.

The other name of the game is to minimize the financial damage by ensuring that the board can be manufactured everywhere and by everyone, in particular those low-cost services.

There are a lot of things to be aware of, and I suggest reading application notes of devices that deal with the relevant type of signals. For example, TI’s board design guidelines for one of their SuperSpeed USB related devices, sllu149e.pdf.

This post focuses on selecting the track width (W) and spacing between the two differential pairs (S) of a coupled microstrip line. This is the widely recommended way to carry RF-frequency differential signals on a PCB, and it simply means that the two wires are routed at one of the exposed layers (top or bottom), and that there’s a ground plane layer immediately underneath it. See drawing at the bottom left of the image below.

Quite interestingly, there is little relation to the W and S of the differential pair in most application notes, so it’s not all that clear what considerations should be made. However, it’s well-known that W and S influence the differential impedance, and this is mentioned everywhere. The USB 3.0 specs requires the differential characteristic impedance to be 90Ω ± 7Ω in section 5.6.1.1.1. So there’s a start.

Assumptions on the PCB

These are the attributes of the most typical PCB one can find (and therefore hopefully low-cost to manufacture):

  • Board material: 1 oz copper (0.035 mm thick) on FR4 resin
  • Minimal track width: 10 mil. I’m generous here, as several low-cost manufacturers offer as low as 4 mil, but 6 mil is often the minimum for the lowest price. Often there’s a remark saying “>8 mil recommended”. So this indicates what’s considered easily manufactured.
  • Minimal spacing between tracks: 10 mils, exactly for the same reasons. Manufacturers tend to give the same figures for both.
  • Vias: I didn’t have any (on the differential pairs, that is).

The PCB calculator

There are plenty of online calculators and other utilities for calculating the impedance of microstrip pairs. Since I used KiCad 5.1.4 for the PCB design, I also went for its PCB calculator (image of screenshot below). It’s not the easiest one however.

KiCad's PCB calculator screenshot on coupled microstrip line(click image to enlarge)

Quite obviously, the board’s parameters are on the left column, and the track’s data to the right. So let’s start with the board’s parameters. Note that the “…” to the right allows selecting materials instead of feeding numbers, and guess what, they’re correct (at least those related to my board).

  • Er: Select FR4, and the dialectric constant is set correctly to 4.6.
  • TanD: Select FR4 here as well, and don’t bother that it’s not accurate (because of the frequency): It doesn’t influence the impedance, but the only the loss calculation.
  • Rho: Select copper. It’s resistivity is correct.
  • H: The height to the ground plane. In my case, it’s a dual-layer board, so the ground plane is the back side. Hence H is the thickness of the entire board, 1 mm.
  • H_t: The height to the ground plane (?) above the board. Luckily, doesn’t have much influence on the impedance unless it’s really small.
  • T: The strip thickness. 0.035 mm for 1 oz.
  • Conductor roughness: Irrelevant for impedance calculation, but will influence conductor loss calculations
  • Relative permeability of conductor: Set to 1 for copper. Copper’s relative permeability, according to Wikipedia, is 0.999994, and this has no influence on impedance calculations.
  • Frequency: For a USB 3.0 link, I set it to 5 GHz. Its influence on the impedance is quite small, so no need to be worried about the idea that a digital signal isn’t really band limited and such.

OK, so now what?

The goal is to find W and S. To the right, there’s also L, the length of the trace. It has no influence on the impedance, so any positive number goes. Likewise, we don’t care about the electrical length (Ang_l) which basically says how much delay, in terms of sinusoidal phase of the given frequency, the conductor imposes.

So there are two input parameters, W and S, and two output parameters, Zeven and Zodd. As the signal is differential with no common mode component, we don’t care about Zeven at all: Since Zdiff = Zodd * 2, we just need to make Zodd = 45Ω, more or less.

Or in other words, there is one degree of freedom. There isn’t one pair of W and S for a given board and impedance, but we get to select one of them.

So the rather silly methodology is to pick some initial W and S, more or less at random, and click the down arrow (or “Analyze”). See what we got. Then set Zodd to 45, and click the up arrow (or “Synthesize”). And then some manual iteration, until the result is fine.

But that leaves us with the need to make a decision.

What is a good pair of W and S?

To make a long story short, as small as possible. These wires aren’t carrying any significant currents, so their size is only limited by the manufacturing process (or: How much we’re ready to pay for it). For low-cost manufacturing, consider that they sometimes make the trace thinner by a mil or two (this is the low-quality side of low-cost). So it’s important to be aware of how this would impact the impedance.

Why is small good? Mainly for crosstalk and noise immunity: The closer the wires are to each other, the more do they behave as a differential pair. Being closer to each other, noise signals are added more equally, leading to a smaller differential noise signal. A common rule of thumb is that a differential pair should have a clearance of 5W from any other conductor.

It’s also a matter of taking up less physical space on the board (in particular when respecting the 5W rule above).

As for signal losses, it doesn’t seem to make a big difference.

There is however no defined way to iterate with the calculator to obtain good values. One may select a minimal W and try out several S until Zodd is 45Ω, but it may be below the minimal S allowed.

In that case, start with the minimal S, and increase W until Zodd is 45Ω. If we want to follow the 5W rule, that is probably the best pair achievable.

As for my own choice, it’s the one shown in the image above: W = 0.8 mm, S = 0.2 mm. This makes S a bit smaller than 8 mils. Even though me original intention was not to go below 10 mils, this settings allows me to meet the 5W rule. And it should still be fine to manufacture virtually anywhere.

These rather bulky tracks weren’t easy to connect to the USB receptacle’s footprint, as discussed in another post.

Other calculators

Given the importance of getting these figures right, it’s reassuring to feed the results into other calculators before etching them in metal.

Note that some calculate Zodd and some give you the differential impedance (plus, possibly some other figures). Don’t get confused, and no panic.

KiCad PCB layout: Adapting the footprint’s pads to match track width

Introduction

It just so happened that I went for a pair of 0.8mm thick wires, with 0.2mm between them, in order to achieve 90Ω differential impedance on a 1 mm thick FR4 board. No wonder PCBnew (KiCad’s layout program) refused connecting these wires to a USB SuperSpeed receptacle’s footprint, with its 0.4 mm side pads, spaced 0.65 mm apart: They simply can’t get squeezed in.

A similar problem occurred with the 0402-sized coupling capacitors in series with one of the differential pairs: Even though there was no problem connecting the differential wires to the capacitors (after reducing the clearance limit for the relevant tracks), the tracks ran over the capacitors’ pads.

This post suggests manipulating the footprints to match the specific width and spacing of a differential pair. Not the easiest way out, but the result is clean and elegant.

The quick and dirty solution is to draw the tracks until close to the footprint, and do the last segment with a thinner track width. Possible, works, but creates ugly discontinuities, and maybe even signal integrity issues due to reflections.

There is more than one way to do it, and I hope it won’t be too confusing as I jump around between the different options. Each has its advantages and disadvantages.

As a side benefit, this post goes into the technique of making arbitrarily shaped pads.

All said below relates to KiCad 5.1.4, which is up to date as of September 2019. On Linux Mint 19, not that it matters.

Footprints, pads, tracks

Before getting into business, let’s talk a bit about pads and how wire tracks are connected to them.

First of all, the footprint: It’s a collection of graphic elements, each assigned to one or more layers, that are plotted on the PCB layout as a group. The relative position between some elements are fixed, and others can be moved (text labels in particular).

A well-known special element is the pad. Aside from the fact that it’s a useful shortcut to draw a shape on several layers at a time, there are three things that are exclusive for pads:

  • Only a pad can be assigned a pin number
  • A pad is the final destination for a wire track (it has an “anchor”).
  • Drill holes (plated or not) can only be defined with pads.

The first two relate to the capability to connect wires to the component’s pin: The pin number, with respect to the netlist, tells PCBnew which wire track to connect to the pad. The anchor tells where to, geometrically, the wire should be connected. Each wire track starts and end at an anchor point. For simple pads, it’s always at their center.

Remember that if you want to draw something on copper or some other layers, and don’t need any of the three exclusives listed above, just drawing filled polygons on the relevant layers might be a better idea.

And now to the wire track’s geometry.

The final product of the PCB layout is Gerber files. Without getting into the details, the way graphics is defined in these files is with straight lines or arcs (part of a circle, or the entire circle), which are given their thickness with an aperture. This aperture is a filled geometric shape, in most cases either a circle or a square with a per-aperture given size.

The circle aperture is apparently dominant in KiCad: When drawing a wire track on a copper layer, it’s effectively a line with a circular aperture. Think of painting the track’s with a marker having a circular shape, with the diameter of the track’s designated thickness.

Actually, it seems like any line thickness in KiCad is implemented with circular aperture. For example, when drawing filled polygons, there are rounded corners when the line thickness is non-zero, due to the circular aperture.

The conclusion of all this: If the pad is circular, with a diameter matching the wire track’s thickness, there will be a perfect overlap between the pad and the track’s end, no matter what direction the track came from. For a rectangular pad, this holds true only if the track comes straight in, and its edge length matches the thickness of the pad in the same direction.

Modifying footprints

We’ll start with the simpler, and less impressive example: The capacitors’ 0402 footprint. The original 0402 footprint looks like this, with 0.59 x 0.64 mm rounded rectangle pads:

Original 0402 footprint

The 0.8 mm thick tracks do connect to a pair of such capacitors, but only after reducing the minimal clearance to 0.15 mm (which is the wrong thing to do):

Tracks coming too close(click image to enlarge)

It’s quite evident that the tracks are all over the pads. This isn’t all that bad, because the solder mask is still correct (not visible on image above). And yet, the tracks get too close to each other, and they’re not supposed to extend the pads on the copper layer.

So this is what an extended 0402 looks like, with the beginning of a 0.8 mm track on each side:

0402 footprint with extensions

Each pad actually consists of three elements: The original pad (without the pad number), a filled polygon and a circular pad, which has the pad number assigned to it.

0402 footprint with extensions, separated

Moving these elements apart a bit for clarity, it looks like this:

0402 footprint with extensions, detailed

But the real need for this technique came from the USB 3.0 receptacle’s footprint, having pads too small for the 0.8 mm tracks:

Micro B receptacle's footprint(click image to enlarge)

To overcome this, extension were added, making pads 0.8 mm wide and 0.2 mm apart. The pad is square in this example, so the track can only arrive directly from above:

Micro B receptacle's footprint, with extensions(click image to enlarge)

Pad 10 of the image above has been separated for the sake of demonstration.

This is what the result looks like in a PCB design using these:

Tracks on PCB(click image to enlarge)

Exaggerated perfectionism or the right thing for a 5 or 10 GHz signal, who knows.

How to do this

A pad with type Custom (having rectangular or circular anchor) can be merged with any number of geometrical shapes, turning it into a larger pad, with the shape of the union of all merged areas.

So:

  • Draw a filled polygon (on the F.Silk for example) and then change its line thickness to 0.
  • Create a pad with “Shape” set to “Custom (Rect. Anchor)”, which will be the actual pad that the wire will be connected to.
  • Select both, right-click, and pick “Create Pad from Selected Shapes”.

The attributes (which layers etc.) are taken from the anchor pad, but the shape is the union of the pad and the polygon. As mentioned above, a circular anchor pad with the diameter of the track width is easiest to work with later (but more difficult to match the polygon for).

In fact, getting the polygons just right is a bit of a headache. I found myself editing the polygons’ coordinates manually in the footprint files to get exact positions. The “Move Exactly” feature is also helpful for placing items accurately.

Note that the extra pad (actually, the anchor pad) should be designated for the copper layer only: No solder mask nor paste layers. We define it as a pad so a track will connect to it, but it’s not going to be soldered on.

What about the courtyard? Should it be extended? Well, no: There’s no problem having other components placed over the extended area, which is part of the wiring. PCBnew ensures no other copper traces get near by virtue of clearance requirements.

Having said all this, note that the trickery with “Create Pad from Selected Shapes” isn’t really necessary. It just makes it simpler to manipulate the shapes as one unit. It also moves the filled polygon from its original layer to the copper layer, but that can be done on the polygon’s own attributes. This way or another, the track connects to the anchor pad (in the merged version) or a simple pad. It doesn’t really matter if the extra geometry is linked to the pad or not.

But now we have two pads

There is the original pad, which is left in place, partly for laziness: It’s already at the right size and place, covering the correct layers (including solder mask and paste), so why mess with it? And then there’s the new pad, which is part of the extension, only for the copper layer.

The question is whether the original pad should also have the pin number, or if it should be pinless. There are basically three options:

  1. Keep the pin number, so both pads have the same number. In this case, PCBnew requires both pads to be connected (wires explicitly with tracks) or the original pad is considered unconnected, and will generate a DRC error. Signing off a PCB layout with unconnected nets errors requires a certain amount of courage. And drawing the internal wire without adding dirt may be tricky.
  2. Remove pin number designation from original pad. But this will cause PCBnew to complain that the new and original pads are too close (in fact, they’re touching). They are considered two different pads, and generate a DRC error like the following:
    ErrType(19): Pad too close to pad
        @(201.775 mm, 57.300 mm): Pad of J1 on F.Cu and others
        @(204.850 mm, 57.950 mm): Pad 9 of J1 on F.Cu

    Well, in some cases this can be solved by skipping the “Create Pad from Selected Shapes” part, and have two (non-custom) pads, with the filled polygon on the copper layer between them, and hence also making enough space between the pads. It’s a question of the size of the polygon vs. the required pad clearance.

  3. Replace the original pad with plain graphics, drawing it as three distinct and overlapping polygons: One for the copper layer, one for the solder mask and one for the solder paste. Rounded corner are achieved with the polygon’s line thickness, if so desired. Haven’t tried this (it’s a bit tedious, after all), but it should work. Note that just merging the original pad’s geometry into the new pad will only cover the copper layer (as the new pad is defined not to cover the two other layers).

I went for the second option, because I allow myself to ignore DRCs on pad-to-pad clearances, but not unconnected net errors. If pad-to-pad clearances are violated because of components placed too close, courtyards are there to prevent them. So making a habit of ignoring pad-to-pad DRC errors is fairly OK.

Impact on signal integrity

I started with this pad manipulation thing because I thought I had no choice, but then I asked myself what this does to the signals. Short answer: I don’t really know, because I haven’t run an electromagnetic simulation on this.

But the word is that any sudden change and discontinuity causes reflections, so connecting to a pad with a gradually changing track width should be better. Maybe, one day, there will be a KiCad feature making all or selected connections between pads and tracks adapting this way.

In fact, think of a thin wire connected to a thicker pad. Wouldn’t it be better if it became gradually thicker as it approaches the pad, so there’s no discontinuity there?

Or maybe it won’t make enough difference to care about this at all. Once again, I don’t know. I guess it’s something the RF guys care more about, and much less us dealing with digital signals.

I should also mention that the way that the pads go sideways on the receptacle shown above, creates a certain difference in the track length, which isn’t accounted for. One can measure the difference manually and feed it into “Pad to die length” in the pad’s attributes, but that was a bit too much for me.

Linux / X-Windows: Which process owns this window?

Once in a while, there’s a piece of junk on the desktop, and the question is who should be blamed for it.

The short answer is:

$ xwininfo

and fetch the window’s ID from the line at the beginning saying e.g.

xwininfo: Window id: 0x860000a "xclock"

And next, fetch the alleged process ID:

$ xprop -id 0x860000a | grep _NET_WM_PID
_NET_WM_PID(CARDINAL) = 58637

Note that it’s the duty of the program that generated the window to set this value correctly, so it may be absent or wrong. If the window belongs to a client on another machine, this process ID might be misleading, as it’s on the process table of the client (check out WM_CLIENT_MACHINE, also given by xprop).

If _NET_WM_PID isn’t helpful, try to look for other hints in xprop’s answer, or the correct process can be found with the rather complicated method described on this page.

Intel Stratix 10 PCIe IP: Building it only appears to be stuck

These are a few notes when implementing an FPGA design for Stratix 10 with a Makefile flow (i.e. command-line utilities) with Quartus 19.2.

So I wanted was to implement a design including the PCIe 16x block for Stratix 10. I added a line going

set_global_assignment -name IP_FILE pcie_s10_16x.ip

in the relevant QSF file. And by the way: Don’t add a Verilog black box module on behalf of this IP in the the design. It will override the IP, and implement an empty module.

Anyhow, attempting to synthesize with quartus_syn failed, saying the RTL files haven’t been generated:

Error (16556): The synthesis RTL for pcie_s10_16x.ip has not been generated. Generate the synthesis RTL from within Platform Designer.

Which was a bit odd. I’m quite used to adding IP files like this, and well, good old quartus_map used to take care of this gracefully. But no.

OK, so I went for building the IP manually. Changed to the relevant directory, and

$ qsys-generate  --synthesis pcie_s10_16x.ip

but that appeared to be stuck. Or more like in some kind of infinite loop. And taking ~ 11 GB of virtual memory with some Java process apparently running forever. But it wasn’t forever. Actually, on a rather strong machine, it eventually ended:

$ time qsys-generate  --synthesis pcie_s10_16x.ip

[ ... ]

real	7m28.275s
user	10m8.443s
sys	0m34.967s

There’s a lot of output there in the middle, but the bottom line is 7 and a half minutes for generating the sources for a PCIe block. Quite amazing. The full transcript is given at the end of this post, for reference.

What apparently seems to take a lot of time is that some preparations are made for each of the PCIe lanes, and there is 16 of them.

The output

So this is what a full session looks like. I’ve cut out the parts that are just repetitions.

$ time qsys-generate  --synthesis pcie_s10_16x.ip
2019.08.16.15:11:30 Warning: Quartus project not specified. Use --quartus-project and --rev to specify a Quartus project and revision.
2019.08.16.15:11:40 Info: Saving generation log to /home/user/try/pcie_s10_16x/pcie_s10_16x_generation.rpt
2019.08.16.15:11:40 Info: Generated by version: 19.2 build 57
2019.08.16.15:11:40 Info: Starting: Create HDL design files for synthesis
2019.08.16.15:11:40 Info: qsys-generate /home/user/try/pcie_s10_16x.ip --synthesis=VERILOG --output-directory=/home/user/try/pcie_s10_16x --family="Stratix 10" --part=1SG280HU2F50E2VG
2019.08.16.15:11:41 Warning: pcie_s10_hip_ast_0: Used altera_pcie_s10_hip_ast 20.0.0 (instead of 18.0)
2019.08.16.15:11:45 Info: pcie_s10_16x.pcie_s10_hip_ast_0: Gen2 (5.0 Gbps) x16 256-bit
2019.08.16.15:11:45 Warning: pcie_s10_16x.pcie_s10_hip_ast_0.rx_st: The empty signal width should be 1 bits.
2019.08.16.15:12:10 Info: pcie_s10_hip_ast_0.fpll_g1g2xn: For the selected device(1SG280HU2F50E2VG), PLL speed grade is 2.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: The TX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: The RX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: For current configuration, TX PCS FIFO depth is "16" and TX Core FIFO depth is "16".
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: For current configuration, RX PCS FIFO depth is "16" and RX Core FIFO depth is "32".
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch0: Note - The external TX PLL IP must be configured with an output clock frequency of 2500.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: The TX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: The RX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: For current configuration, TX PCS FIFO depth is "16" and TX Core FIFO depth is "16".
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: For current configuration, RX PCS FIFO depth is "16" and RX Core FIFO depth is "32".
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:12:11 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch1: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.

[ The same over and over again... ]

2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: The TX PCS-Core Interface FIFO is operating in half-rate transfer mode.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: The RX PCS-Core Interface FIFO is operating in half-rate transfer mode.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: For current configuration, TX PCS FIFO depth is "8" and TX Core FIFO depth is "8".
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: For current configuration, RX PCS FIFO depth is "8" and RX Core FIFO depth is "16".
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.
2019.08.16.15:12:24 Info: phy_g2x16.altera_xcvr_pcie_hip_channel_s10_ch15: Note - The external TX PLL IP must be configured with an output clock frequency of 2500.0 MHz.
2019.08.16.15:12:24 Info: pcie_s10_16x: "Transforming system: pcie_s10_16x"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Naming system components in system: pcie_s10_16x"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Processing generation queue"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Generating: pcie_s10_16x"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Generating: altera_pcie_s10_hip_ast"
2019.08.16.15:12:24 Info: phy_g2x16: "Generating: phy_g2x16"
2019.08.16.15:12:24 Info: fpll_g1g2xn: "Generating: fpll_g1g2xn"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Generating: phy_g2x16"
2019.08.16.15:12:24 Info: pcie_s10_16x: "Generating: fpll_g1g2xn"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_wqso57y"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: "Generating: altera_xcvr_pcie_hip_channel_s10_ch0"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch1: "Generating: altera_xcvr_pcie_hip_channel_s10_ch1"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch2: "Generating: altera_xcvr_pcie_hip_channel_s10_ch2"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch3: "Generating: altera_xcvr_pcie_hip_channel_s10_ch3"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch4: "Generating: altera_xcvr_pcie_hip_channel_s10_ch4"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch5: "Generating: altera_xcvr_pcie_hip_channel_s10_ch5"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch6: "Generating: altera_xcvr_pcie_hip_channel_s10_ch6"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch7: "Generating: altera_xcvr_pcie_hip_channel_s10_ch7"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch8: "Generating: altera_xcvr_pcie_hip_channel_s10_ch8"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch9: "Generating: altera_xcvr_pcie_hip_channel_s10_ch9"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch10: "Generating: altera_xcvr_pcie_hip_channel_s10_ch10"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch11: "Generating: altera_xcvr_pcie_hip_channel_s10_ch11"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch12: "Generating: altera_xcvr_pcie_hip_channel_s10_ch12"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch13: "Generating: altera_xcvr_pcie_hip_channel_s10_ch13"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch14: "Generating: altera_xcvr_pcie_hip_channel_s10_ch14"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch15: "Generating: altera_xcvr_pcie_hip_channel_s10_ch15"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_fpll_s10_htile_1911_xs4xrjq"
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./altera_std_synchronizer_nocut.v VERILOG PATH ../../../../primitives/altera_std_synchronizer/altera_std_synchronizer_nocut.v
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./s10_avmm_h.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_native_phy/altera_xcvr_native_s10/source/s10_avmm_h.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./altera_xcvr_native_s10_functions_h.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_native_phy/altera_xcvr_native_s10/source/altera_xcvr_native_s10_functions_h.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_native_anlg_reset_seq.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_native_phy/altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_pll_rcfg_arb.sv SYSTEM_VERILOG PATH ../../common/alt_xcvr_pll_rcfg_arb.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_pll_embedded_debug.sv SYSTEM_VERILOG PATH ../../common/alt_xcvr_pll_embedded_debug.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_pll_avmm_csr.sv SYSTEM_VERILOG PATH ../../common/alt_xcvr_pll_avmm_csr.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_resync.sv SYSTEM_VERILOG PATH ../../../../altera_xcvr_generic/ctrl/alt_xcvr_resync.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: add_fileset_file ./alt_xcvr_arbiter.sv SYSTEM_VERILOG PATH ../../../../altera_xcvr_generic/ctrl/alt_xcvr_arbiter.sv
2019.08.16.15:12:25 Info: fpll_g1g2xn: Building configuration data for reconfiguration profile 0
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_3e57toy"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_zj3agaa"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_prgcfha"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_jgm5cfa"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_ykdbt6y"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_lzoehjq"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_r73nlzi"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_3qbzhci"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_4b2ohhy"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_45l4xnq"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_ipejypi"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_jtr36ma"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_dfssemq"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_dtiglpy"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_lh3sg5q"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_pcie_hip_native_s10_191_zgzdh3a"
2019.08.16.15:12:25 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq"
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_arbiter.sv SYSTEM_VERILOG PATH ../../../../altera_xcvr_generic/ctrl/alt_xcvr_arbiter.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./altera_std_synchronizer_nocut.v VERILOG PATH ../../../../primitives/altera_std_synchronizer/altera_std_synchronizer_nocut.v
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_resync_std.sv SYSTEM_VERILOG PATH ../../../alt_xcvr_generic/alt_xcvr_resync_std.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_reset_counter_s10.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_reset_control_s10/alt_xcvr_reset_counter_s10.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./altera_xcvr_native_s10_functions_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/altera_xcvr_native_s10_functions_h.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./s10_avmm_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/s10_avmm_h.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_avmm_csr.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_avmm_csr.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_prbs_accum.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_prbs_accum.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_odi_accel.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_odi_accel.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_rcfg_arb.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rcfg_arb.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_early_spd_chng_s10_htile.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_early_spd_chng_s10_htile.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_anlg_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_dig_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_dig_reset_seq.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_reset_seq.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_anlg_reset_seq_wrapper.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq_wrapper.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_re_cal_chnl.v VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_re_cal_chnl.v
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_pcie_rx_eios_prot.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_pcie_rx_eios_prot.sv
2019.08.16.15:12:25 Info: altera_xcvr_pcie_hip_channel_s10_ch0: add_fileset_file ./alt_xcvr_native_rx_maib_wa.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rx_maib_wa.sv
2019.08.16.15:12:26 Info: altera_xcvr_pcie_hip_channel_s10_ch0: Building configuration data for reconfiguration profile 0
ip-deploy --component-name=altera_xcvr_native_s10_htile --output-name=pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq --output-directory=/tmp/alt8124_4770953449535589358.dir/0005_altera_xcvr_pcie_hip_channel_s10_ch0_gen --component-param=rcfg_enable=1 --component-param=device=1SG280HU2F50E2VG --component-param=base_device=ND5U {--component-param=device_die_types=HSSI_CRETE2E MAIN_ND5} {--component-param=device_die_revisions=HSSI_CRETE2E_REVB MAIN_ND5_REVC} --component-param=protocol_mode=pipe_g2 --component-param=set_data_rate=5000 --component-param=bonded_mode=pma_pcs --component-param=pcs_reset_sequencing_mode=bonded --component-param=enable_manual_bonding_settings=1 --component-param=manual_pcs_bonding_mode=ctrl_slave_blw --component-param=manual_pcs_bonding_comp_cnt=2 --component-param=manual_tx_hssi_aib_bonding_mode=ctrl_master_bot --component-param=manual_tx_hssi_aib_bonding_comp_cnt=6 --component-param=manual_tx_core_aib_bonding_mode=ctrl_master_bot --component-param=manual_tx_core_aib_bonding_comp_cnt=6 --component-param=manual_tx_hssi_aib_indv=indv_dis --component-param=manual_tx_core_aib_indv=indv_dis --component-param=set_cdr_refclk_freq=100.000 --component-param=enable_hip=1 --component-param=enable_hard_reset=1 --component-param=set_hip_cal_en=1 --component-param=hip_mode=user_chnl --component-param=hip_prot_mode=gen2 --component-param=hip_channels=x16 {--component-param=std_tx_byte_ser_mode=Serialize x4} {--component-param=std_rx_byte_deser_mode=Deserialize x4} --component-param=std_tx_8b10b_enable=1 --component-param=std_tx_8b10b_disp_ctrl_enable=1 --component-param=std_rx_8b10b_enable=1 --component-param=std_rx_rmfifo_mode=pipe --component-param=std_rx_rmfifo_pattern_n=192892 --component-param=std_rx_rmfifo_pattern_p=855683 {--component-param=std_rx_word_aligner_mode=synchronous state machine} --component-param=std_rx_word_aligner_pattern_len=10 --component-param=std_rx_word_aligner_pattern=380 --component-param=std_rx_word_aligner_renumber=16 --component-param=std_rx_word_aligner_rgnumber=15 --component-param=enable_ports_pipe_sw=1 --component-param=enable_ports_pipe_rx_elecidle=1 --component-param=enable_ports_pipe_hclk=1 --component-param=tx_fifo_pfull=10 --component-param=rx_fifo_pfull=10 --component-param=enable_port_tx_clkout2=1 --component-param=tx_clkout2_sel=pcs_x2_clkout --component-param=qsf_assignments_enable=1 --component-param=rx_pma_term_sel=r_r4
***************************************************************
Quartus is a registered trademark of Intel Corporation in the
US and other countries.  Portions of the Quartus Prime software
code, and other portions of the code included in this download
or on this DVD, are licensed to Intel Corporation and are the
copyrighted property of third parties. For license details,
refer to the End User License Agreement at

http://fpgasoftware.intel.com/eula.

***************************************************************

2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: The TX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: The RX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: For current configuration, TX PCS FIFO depth is "16" and TX Core FIFO depth is "16".
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: For current configuration, RX PCS FIFO depth is "16" and RX Core FIFO depth is "32".
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.
2019.08.16.15:12:48 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq: Note - The external TX PLL IP must be configured with an output clock frequency of 2500.0 MHz.
2019.08.16.15:12:48 Info: Deploying pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq to /tmp/alt8124_4770953449535589358.dir/0005_altera_xcvr_pcie_hip_channel_s10_ch0_gen/pcie_s10_16x_altera_xcvr_native_s10_htile_1920_3nhmcmq.ip
2019.08.16.15:12:49 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y"
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_arbiter.sv SYSTEM_VERILOG PATH ../../../../altera_xcvr_generic/ctrl/alt_xcvr_arbiter.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./altera_std_synchronizer_nocut.v VERILOG PATH ../../../../primitives/altera_std_synchronizer/altera_std_synchronizer_nocut.v
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_resync_std.sv SYSTEM_VERILOG PATH ../../../alt_xcvr_generic/alt_xcvr_resync_std.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_reset_counter_s10.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_reset_control_s10/alt_xcvr_reset_counter_s10.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./altera_xcvr_native_s10_functions_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/altera_xcvr_native_s10_functions_h.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./s10_avmm_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/s10_avmm_h.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_avmm_csr.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_avmm_csr.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_prbs_accum.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_prbs_accum.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_odi_accel.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_odi_accel.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_rcfg_arb.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rcfg_arb.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_early_spd_chng_s10_htile.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_early_spd_chng_s10_htile.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_anlg_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_dig_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_dig_reset_seq.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_reset_seq.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_anlg_reset_seq_wrapper.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq_wrapper.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_re_cal_chnl.v VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_re_cal_chnl.v
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_pcie_rx_eios_prot.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_pcie_rx_eios_prot.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: add_fileset_file ./alt_xcvr_native_rx_maib_wa.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rx_maib_wa.sv
2019.08.16.15:12:49 Info: altera_xcvr_pcie_hip_channel_s10_ch1: Building configuration data for reconfiguration profile 0
ip-deploy --component-name=altera_xcvr_native_s10_htile --output-name=pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y --output-directory=/tmp/alt8124_4770953449535589358.dir/0006_altera_xcvr_pcie_hip_channel_s10_ch1_gen --component-param=rcfg_enable=1 --component-param=device=1SG280HU2F50E2VG --component-param=base_device=ND5U {--component-param=device_die_types=HSSI_CRETE2E MAIN_ND5} {--component-param=device_die_revisions=HSSI_CRETE2E_REVB MAIN_ND5_REVC} --component-param=protocol_mode=pipe_g2 --component-param=set_data_rate=5000 --component-param=bonded_mode=pma_pcs --component-param=pcs_reset_sequencing_mode=bonded --component-param=enable_manual_bonding_settings=1 --component-param=manual_pcs_bonding_mode=ctrl_slave_blw --component-param=manual_pcs_bonding_comp_cnt=4 --component-param=manual_tx_hssi_aib_bonding_mode=ctrl_slave_abv --component-param=manual_tx_hssi_aib_bonding_comp_cnt=4 --component-param=manual_tx_core_aib_bonding_mode=ctrl_slave_abv --component-param=manual_tx_core_aib_bonding_comp_cnt=4 --component-param=manual_tx_hssi_aib_indv=indv_dis --component-param=manual_tx_core_aib_indv=indv_dis --component-param=set_cdr_refclk_freq=100.000 --component-param=enable_hip=1 --component-param=enable_hard_reset=1 --component-param=set_hip_cal_en=1 --component-param=hip_mode=user_chnl --component-param=hip_prot_mode=gen2 --component-param=hip_channels=x16 {--component-param=std_tx_byte_ser_mode=Serialize x4} {--component-param=std_rx_byte_deser_mode=Deserialize x4} --component-param=std_tx_8b10b_enable=1 --component-param=std_tx_8b10b_disp_ctrl_enable=1 --component-param=std_rx_8b10b_enable=1 --component-param=std_rx_rmfifo_mode=pipe --component-param=std_rx_rmfifo_pattern_n=192892 --component-param=std_rx_rmfifo_pattern_p=855683 {--component-param=std_rx_word_aligner_mode=synchronous state machine} --component-param=std_rx_word_aligner_pattern_len=10 --component-param=std_rx_word_aligner_pattern=380 --component-param=std_rx_word_aligner_renumber=16 --component-param=std_rx_word_aligner_rgnumber=15 --component-param=enable_ports_pipe_sw=1 --component-param=enable_ports_pipe_rx_elecidle=1 --component-param=enable_ports_pipe_hclk=1 --component-param=tx_fifo_pfull=10 --component-param=rx_fifo_pfull=10 --component-param=enable_port_tx_clkout2=1 --component-param=tx_clkout2_sel=pcs_x2_clkout --component-param=qsf_assignments_enable=1 --component-param=rx_pma_term_sel=r_r4
***************************************************************
Quartus is a registered trademark of Intel Corporation in the
US and other countries.  Portions of the Quartus Prime software
code, and other portions of the code included in this download
or on this DVD, are licensed to Intel Corporation and are the
copyrighted property of third parties. For license details,
refer to the End User License Agreement at

http://fpgasoftware.intel.com/eula.

***************************************************************

2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: The TX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: The RX PCS-Core Interface FIFO is operating in full-rate transfer mode.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: For current configuration, TX PCS FIFO depth is "16" and TX Core FIFO depth is "16".
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: For current configuration, RX PCS FIFO depth is "16" and RX Core FIFO depth is "32".
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.
2019.08.16.15:13:12 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y: Note - The external TX PLL IP must be configured with an output clock frequency of 2500.0 MHz.
2019.08.16.15:13:12 Info: Deploying pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y to /tmp/alt8124_4770953449535589358.dir/0006_altera_xcvr_pcie_hip_channel_s10_ch1_gen/pcie_s10_16x_altera_xcvr_native_s10_htile_1920_prfyb7y.ip
2019.08.16.15:13:13 Info: pcie_s10_16x: "Generating: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_cqbemta"
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_arbiter.sv SYSTEM_VERILOG PATH ../../../../altera_xcvr_generic/ctrl/alt_xcvr_arbiter.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./altera_std_synchronizer_nocut.v VERILOG PATH ../../../../primitives/altera_std_synchronizer/altera_std_synchronizer_nocut.v
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_resync_std.sv SYSTEM_VERILOG PATH ../../../alt_xcvr_generic/alt_xcvr_resync_std.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_reset_counter_s10.sv SYSTEM_VERILOG PATH ../../../altera_xcvr_reset_control_s10/alt_xcvr_reset_counter_s10.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./altera_xcvr_native_s10_functions_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/altera_xcvr_native_s10_functions_h.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./s10_avmm_h.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/s10_avmm_h.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_avmm_csr.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_avmm_csr.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_prbs_accum.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_prbs_accum.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_odi_accel.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_odi_accel.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_rcfg_arb.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rcfg_arb.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_early_spd_chng_s10_htile.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_early_spd_chng_s10_htile.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_anlg_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_dig_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_dig_reset_seq.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_reset_seq.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_reset_seq.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_anlg_reset_seq_wrapper.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_anlg_reset_seq_wrapper.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_re_cal_chnl.v VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_re_cal_chnl.v
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_pcie_rx_eios_prot.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_pcie_rx_eios_prot.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: add_fileset_file ./alt_xcvr_native_rx_maib_wa.sv SYSTEM_VERILOG PATH ../../altera_xcvr_native_s10/source/alt_xcvr_native_rx_maib_wa.sv
2019.08.16.15:13:13 Info: altera_xcvr_pcie_hip_channel_s10_ch2: Building configuration data for reconfiguration profile 0
ip-deploy --component-name=altera_xcvr_native_s10_htile --output-name=pcie_s10_16x_altera_xcvr_native_s10_htile_1920_cqbemta --output-directory=/tmp/alt8124_4770953449535589358.dir/0007_altera_xcvr_pcie_hip_channel_s10_ch2_gen --component-param=rcfg_enable=1 --component-param=device=1SG280HU2F50E2VG --component-param=base_device=ND5U {--component-param=device_die_types=HSSI_CRETE2E MAIN_ND5} {--component-param=device_die_revisions=HSSI_CRETE2E_REVB MAIN_ND5_REVC} --component-param=protocol_mode=pipe_g2 --component-param=set_data_rate=5000 --component-param=bonded_mode=pma_pcs --component-param=pcs_reset_sequencing_mode=bonded --component-param=enable_manual_bonding_settings=1 --component-param=manual_pcs_bonding_mode=ctrl_slave_blw --component-param=manual_pcs_bonding_comp_cnt=6 --component-param=manual_tx_hssi_aib_bonding_mode=ctrl_slave_abv --component-param=manual_tx_hssi_aib_bonding_comp_cnt=2 --component-param=manual_tx_core_aib_bonding_mode=ctrl_slave_abv --component-param=manual_tx_core_aib_bonding_comp_cnt=2 --component-param=manual_tx_hssi_aib_indv=indv_dis --component-param=manual_tx_core_aib_indv=indv_dis --component-param=set_cdr_refclk_freq=100.000 --component-param=enable_hip=1 --component-param=enable_hard_reset=1 --component-param=set_hip_cal_en=1 --component-param=hip_mode=user_chnl --component-param=hip_prot_mode=gen2 --component-param=hip_channels=x16 {--component-param=std_tx_byte_ser_mode=Serialize x4} {--component-param=std_rx_byte_deser_mode=Deserialize x4} --component-param=std_tx_8b10b_enable=1 --component-param=std_tx_8b10b_disp_ctrl_enable=1 --component-param=std_rx_8b10b_enable=1 --component-param=std_rx_rmfifo_mode=pipe --component-param=std_rx_rmfifo_pattern_n=192892 --component-param=std_rx_rmfifo_pattern_p=855683 {--component-param=std_rx_word_aligner_mode=synchronous state machine} --component-param=std_rx_word_aligner_pattern_len=10 --component-param=std_rx_word_aligner_pattern=380 --component-param=std_rx_word_aligner_renumber=16 --component-param=std_rx_word_aligner_rgnumber=15 --component-param=enable_ports_pipe_sw=1 --component-param=enable_ports_pipe_rx_elecidle=1 --component-param=enable_ports_pipe_hclk=1 --component-param=tx_fifo_pfull=10 --component-param=rx_fifo_pfull=10 --component-param=enable_port_tx_clkout2=1 --component-param=tx_clkout2_sel=pcs_x2_clkout --component-param=qsf_assignments_enable=1 --component-param=rx_pma_term_sel=r_r4

[ The same over and over again... ]

***************************************************************
Quartus is a registered trademark of Intel Corporation in the
US and other countries.  Portions of the Quartus Prime software
code, and other portions of the code included in this download
or on this DVD, are licensed to Intel Corporation and are the
copyrighted property of third parties. For license details,
refer to the End User License Agreement at

http://fpgasoftware.intel.com/eula.

***************************************************************

2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: The TX PCS-Core Interface FIFO is operating in half-rate transfer mode.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: The RX PCS-Core Interface FIFO is operating in half-rate transfer mode.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: For the selected device(1SG280HU2F50E2VG), transceiver speed grade is 2 and core speed grade is 2.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: For current configuration, TX PCS FIFO depth is "8" and TX Core FIFO depth is "8".
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: For current configuration, RX PCS FIFO depth is "8" and RX Core FIFO depth is "16".
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: "PCS clkout" (pcs_clkout) is selected to drive tx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: "PCS clkout x2" (pcs_x2_clkout) is selected to drive tx_clkout2 port and the clock frequency is 0.0 MHz.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: "PCS clkout" (pcs_clkout) is selected to drive rx_clkout port and the clock frequency is 0.0 MHz.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: A design example cannot be generated for "Tx PLL reference clock frequency"=="125.0" && "Selected CDR reference clock frequency"=="100.000"
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: Design example cannot be generated for the current configuration. Please check under Design Example tab and system message below for more detailed information.
2019.08.16.15:18:55 Info: pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i: Note - The external TX PLL IP must be configured with an output clock frequency of 2500.0 MHz.
2019.08.16.15:18:55 Info: Deploying pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i to /tmp/alt8124_4770953449535589358.dir/0020_altera_xcvr_pcie_hip_channel_s10_ch15_gen/pcie_s10_16x_altera_xcvr_native_s10_htile_1920_esxl67i.ip
2019.08.16.15:18:56 Info: pcie_s10_16x: Done "pcie_s10_16x" with 38 modules, 428 files
2019.08.16.15:18:57 Info: qsys-generate succeeded.
2019.08.16.15:18:57 Info: Finished: Create HDL design files for synthesis
***************************************************************
Quartus is a registered trademark of Intel Corporation in the
US and other countries.  Portions of the Quartus Prime software
code, and other portions of the code included in this download
or on this DVD, are licensed to Intel Corporation and are the
copyrighted property of third parties. For license details,
refer to the End User License Agreement at

http://fpgasoftware.intel.com/eula.

***************************************************************
real	7m28.275s
user	10m8.443s
sys	0m34.967s

Intel / Altera: Proper access of the Configuration Space Registers (tl_cfg_ctl)

Introduction

The PCIe blocks on Intel (formerly Altera) FPGAs have a somewhat peculiar, yet useful interface for exposing some of the interface’s configuration information, called “Transaction Layer Configuration”. For the purposes of this post, it consists of two output signals, tl_cfg_add[3:0] and tl_cfg_ctl[31:0]. Both are driven by the PCIe block, and clocked by the same clock. The documentation seems to be a bit confused on whether this clock is pld_clk or coreclkout_hip however (even in different versions of the same user guide).

Recall that coreclkout_hip is driven by the PCIe block, and pld_clk is an input to it. The straightforward choice is to connect the pld_clk input to coreclkout_hip, so the PCIe block, as well as the application logic, are both driven by the clock the PCIe block generates. It’s actually not clear to me why anyone would go for a different solution, but the user guide keeps this option open.

So I’ll assume pld_clk and coreclkout_hip are the same signal here. Hence the said confusion in the documentation makes no difference. However it may be hint to why Intel changed the guidelines to interacting with the tl_cfg_add / tl_cfg_ctl pair, probably around the release of Quartus 16.0.

The interface in brief

It’s more or less like a cyclic slide show: tl_cfg_add is incremented every 4th or 8th clock (i.e. pld_clk = coreclkout_hip), and tl_cfg_ctl contains a value that corresponds to the position in some register array, at the address shown in tl_cfg_add. Both change on the same rising edge of the same clock.

This register array appears in the relevant FPGA family’s user guide, has no relation to anything standard, and varies from one FPGA family to another. The data is packed into this 16-word register array in a creative manner at times. The information that can be obtained on an endpoint PCIe interface is in particular the bus address, Max_Payload_Size, Max_Read_Request_Size and if the RCB is 128 bytes. On post-Cyclone IV devices, the data for producing an MSI interrupt can be obtained from this interface as well.

The old API

In older revisions of the user guides (and even the current revision for Cyclone IV), the guideline is to treat tl_cfg_add and tl_cfg_ctl as synchronous signals, so it made perfect sense to go (on Cyclone 10 and several others)

always @(posedge pld_clk)
  case (tl_cfg_add)
    0: cfg_dcommand <= tl_cfg_ctl[31:16];
    2: cfg_lcommand <= tl_cfg_ctl[31:16];
  endcase

The fact that tl_cfg_add dwells a few clocks on each address makes no difference, as tl_cfg_ctl contains the same, correct, value on all of these clock cycles.

The update

In more recent user guides issued by Intel, in particular for Series V and Series 10 FPGAs, the guidelines have changed. Apparently, the timing constraints were not properly applied before Quartus 16.0.1, and one can’t treat tl_cfg_add and tl_cfg_ctl as synchronous signals anymore. Rather, the guidelines require that application logic detects the change in tl_cfg_add’s least significant bit, and then sample both signals with a safe time margin. Without saying it explicitly, the user guides treat these two as signals from an unrelated clock domain.

The fact that both signals remain constant for a fixed number of clock cycles makes it possible to write simple logic that ensures proper sampling.

From a practical point of view, I can testify that the old API works well regardless of the new, stricter, guidelines. It’s not clear whether the sampling mechanism is actually required for proper operation, or if it’s a leftover in the guidelines for Quartus revisions that didn’t enforce the timing constraints properly on the said signals. This way or another, if the user guide requires something, do it. Nevertheless, odds are that FPGA designs already out there, based upon the old API, are still fine.

The new guideline

In short, updated user guides require that the tl_cfg_add and tl_cfg_ctl are sampled in the middle of their time window with stable values. The beginning of each such time window is detected by the application logic by a change in the least significant bit of tl_cfg_add.

There are however a few things to note:

  • The user guides state that the said time window is either 4 clocks or 8 clocks, “depending on the parameterization”, but don’t say how to tell which one applies for a given design.
  • As tl_cfg_add is considered an asynchronous signal, detecting changes on its least significant bit must be done on a register that samples it on each clock, and not comparing directly with tl_cfg_add[0].
  • It’s not possible to sample in the middle of a time window consisting of an even number of clock cycles. In the user guide’s example, the sampling is timed at the clock cycle after the middle of the time window.

The user guide shows sample Verilog code for a clock window of 8 clocks, but doesn’t relate to the 4 clock case.

Verilog code

The following Verilog code can be used to implement the generation of the sampling strobe:

reg [3:0] tl_cfg_add0_d;
reg       strobe;
reg [3:0] counter, prev_counter;

always @(posedge pld_clk)
  begin
    tl_cfg_add0_d <= { tl_cfg_add0_d, tl_cfg_add[0] };

    if (prev_counter > 5)
      strobe <= (tl_cfg_add0_d[2] != tl_cfg_add0_d[3]);
    else
      strobe <= (tl_cfg_add0_d[0] != tl_cfg_add0_d[1]);

    if (tl_cfg_add0_d[0] == tl_cfg_add0_d[1])
      counter <= counter + 1;
    else
      begin
        prev_counter <= counter;
        counter <= 0;
      end
  end

and then this strobe can be used to sample the two signals:

always @(posedge pld_clk)
  if (strobe)
    begin
      tl_cfg_add_samp <= tl_cfg_add;
      tl_cfg_ctl_samp <= tl_cfg_ctl;
    end

The idea is to detect whether the time window is 4 or 8 clock cycles by counting them, and storing the value just before resetting the counter back to zero in prev_counter. For a 4 clock window, it’s expected to be 4, but may also turn out 3 or 5 due to momentary timing glitches. Likewise, prev_counter may turn out either 7, 8 or 9 when an 8 clock window is in effect.

prev_counter is used to select when to assert strobe: If it’s larger than 5, a timing suitable for an 8-clock window is selected. If not, the timing for a 4 clock window. Even though prev_counter may fluctuate from one window to another, it’s not expected to change in a way that alters the selection. Therefore, the fact that prev_counter was measured on one time window, and is used to time the sampling of a another, has no significance. It might as well have been measured on one time window and applied forever afterwards, but that would have required logic that determines when it’s valid.

The sampling instance for a 4-clock time window is two clocks after the change in tl_cfg_add[0], which is consistent with the guideline to sample at the middle of the window (actually, on the closest clock cycle after the middle). For an 8-clock cycle, the delay is 4 clocks, exactly as demonstrated in the examples in the user guides.

Finally, the sampled data can be consumed, e.g.

always @(posedge pld_clk)
  begin
    strobe_d <= strobe;

    if (strobe_d)
      case (tl_cfg_add_samp)
         0: cfg_dcommand <= tl_cfg_ctl_samp[31:16];
         2: cfg_lcommand <= tl_cfg_ctl_samp[31:16];
      endcase
  end

Note that strobe_d is used rather than strobe for consuming that sampled values (even though it would work likewise with strobe, just with a slight delay of the update).