my tech blog

Vintage Tektronix 515A oscilloscope is looking for a new home

Tektronix 515A Oscilloscope

I got it when I was 13. Yes, some of us play with strange toys. I had a lot of fun moments with it, but the truth is that even if it worked, it’s not much of use, now that I’ve betrayed it with a young and colorful one.

Mechanically, this old scope is in good shape (it’s the one in the picture above). Electronically, it does turn on, but there is no beam. The panel lamps behave as expected, so I suppose that with some love and care, it can be fixed without major surgery.

The scope was manufactured in 1957. Naturally, it’s based upon vacuum tubes, which gives it the same nostalgic feel as an old radio.

There is no point in myself keeping it, but my sentiments won’t let me throw it in the dustbin. So I’m ready to give it away to a museum or serious collector. My basic condition, is that whoever wants it, will have to pay for the shipping. And if you’re serious about this kind of gear, you surely know that this “portable scope” weights 20 kg.

In case you’re interested, please drop me a note. My email address can be found at my website.

June 2019 update: Nah, I’ll keep it. If it’s been with me this long, it was probably meant to stay with me.

CSS and DIV layout: The main pitfalls

In the beginning web pages were simple and innocent. Then people started to abuse tables to achieve some layout. And then came CSS and DIV, the winning combination for making a web page oh-so-beautiful but not resizable. Web designers started telling the browser where to put what, and the browser responded with blind obedience, even if the result was a rubble of text and graphics. That’s the deal: Use absolute positioning to get full control, but lose the browser’s capability of avoiding nasty accidents.

So here is my short list of the main problems you may run into, while attempting to layout a web page using CSS-backed DIVs. If you really have to, that is.

Pitfall #0: Resize

If your page is made with CSS, don’t let the width change. Period. Play with CSS/DIV layout only when you’re inside a box, whose width is known and fixed to the pixel. This is not very browser tolerant, I know. It may not work well on a tiny screen. Or a very big one.

But trust me on this: If you don’t set the overall width, you will sooner or later find yourself with spillovers and overlapping graphics. Maybe you can get it to survive a page resize on your browser. You have no chance with the IE version still not out.

If you want to allow a resize, use tables.

Pitfall #1: Huge CSS file (reference by ID)

This one is so common, that sometimes I wonder if I missed something.

Anyhow, the original idea of CSS, if I got it right, was to define the format in one place and put the content in another. I suppose someone hoped that people would go on using simple tags such as <h1> and <p> and get beautiful pages with clean HTML.

And to make things nicer, they allowed classes. So if you wanted a special paragraph, you just went ‘p.special’ in the style definition, and then <p class=”special”> in the HTML, and there you were, the HTML still readable and the format consistent and depending on the content.

But then came style-by-ID, which turned everything into a joke. This made it possible to define the style for a very certain element, pin-pointing it by its ID tag. So if I wanted to format one single paragraph, I’d go ‘#theID’ in the CSS file, and then <p id=”theID”> in the HTML. And that makes me wonder: If the styling is going to be used once anyhow, why not put it directly in the HTML? Why not go <p style=”…”> instead? Don’t tell me that you separate formatting from content, when you can’t change anything in the CSS file without checking the specific place where it’s used. If you remember where it was, that is.

Pitfall #2: The IE ‘width’ bug

The CSS2 spec paragraph 10.2 clearly defines the ‘width’ property as “that of the rendered content within”. Microsoft probably said “we are the standard” and decided to include the padding and the border in the ‘width’ property. This is said to be fixed in IE6, so maybe it isn’t an issue today, after all.

Below is drawing of the box model, and the difference between the two definitions of ‘width’. The sad bottom line is that if you need a consistent width of some element, padding and border have to be zero. If this is not an option, the only safe solution is to put your element in another element (a DIV, I suppose) and define the ‘width’ for the enclosing element.

Box model

Pitfall #3: Getting absolute positioning wrong

So let’s go through a few quick facts:

• Absolute positioning puts your element where it would anyhow, unless you specify otherwise for the X axis (with “left” and “right” style properties) and/or the Y axis (with “top” or  “bottom” style properties). The only difference is that no space is allocated on the rendered page. The responsibility to keep the underlying area clean (or intentionally overlapping) is all yours.  Usually, if you don’t know exactly what you’re doing, things will get wrong sooner or later.
• If you want the element on a fixed position on the screen, no matter how the page is scrolled (logo?), used the “fixed” position.
• “relative” position is not what it appears to be. It will move the block from its original position, but allocate  space for it, as if it was there. Makes sense if you want chemistry-style subscripting, but otherwise I can see only one use: The zero-offset relative positioning, which is a way to set the ‘position’ property to something else than static (the default) without actually doing anything, and hence create a “containing block” in its natural place (more about that next).
• The “left”, “right”, “top” and “bottom” properties are related to the “containing block”, whose definition is a pitfall in itself. (See the CSS2 spec, paragraph 10.1, item 4). To make a long story short, the browser will climb up the hierarchy of blocks until it finds one, which isn’t statically positioned. It will then respect that block’s padding, and there is the zero point.
  Now, “statically positioned” is the default setting of the “position” property, which means that the browser should put the element following the “old-fashioned” rules. See the example below for how to use the “relative” position to make the enclosing DIV effective.
  If the containing block isn’t explicitly defined, hell breaks loose. The browser will most likely pick the root block, which is the top left corner of the page. Or not, depending on other junk you have on your page. Everything happens.

Pitfall #4: Absolute positioning know-it-all

I think that the main reason why “professional” web designers started talking about replacing <TABLE> formatting with <DIV> and CSS was that they first painted their web page with some graphics tool, and then wanted to copy their graphic design exactly into the web page. Which is a nice concept, if you can assure that the page will be viewed exactly like you expect: The same fonts, the same sizes, the same screen size, and so on. Which you can’t, but that’s another story.

No wonder they hated tables. It’s really a headache to control how the browser displays tables down to the pixel. Tables were, after all, not meant to be used that way. Getting the graphics from one table cell aligned with the next one is not easy. Not to mention more complex relations between graphic elements. And then came CSS and the DIVs.

The real power with DIVs is absolute positioning. Just make yourself a box, decide how large it will be (down to the pixel) and put whatever you want inside, exactly where you want it. Something like this:

<div style="position: relative; height: 200px; width: 200px; border: dotted red 1px;">
This is my canvas!
<div style="position: absolute; top: 60px; left: 20px">And</div>
<div style="position: absolute; top: 50px; left: 70px">I</div>
<div style="position: absolute; top: 55px; left: 90px">can</div>
<div style="position: absolute; top: 60px; left: 130px">put</div>
<div style="position: absolute; top: 110px; left: 40px">the</div>
<div style="position: absolute; top: 105px; left: 70px">elements</div>
<div style="position: absolute; top: 112px; left: 140px">where</div>
<div style="position: absolute; top: 150px; left: 80px">I</div>
<div style="position: absolute; top: 155px; left: 90px">want!</div>
</div>

And you get this:

Now I’m asking, isn’t this heaven? I say: It is, on your browser, on your computer. But since you’ve told the browser exactly what to do, you’ve also crippled its ability to fix things when the page is shown differently from how you expect it. A simple example follows.

Pitfall #5: Overflow

The page got resized? The fonts are shown larger than you defined? There was more text than expected? In the old days, the browser used to fix this. With CSS and DIVs you get rubbish.

The CSS spec says, that any block can overflow. What to do with the excessive graphic pulp is up to the browser (or it can be defined) but no matter how it’s handled, the result is ugly.

Let’s take a silly menu layout for example:

<div style="position: relative; width: 120px; height: 70px; border: solid 1px black">
<div style="position: absolute; top: 10px; left: 20px; background-color: #f8f;">Take me!</div>
<div style="position: absolute; top: 40px; left: 20px; background-color: #ff8;">Or take me!</div>
</div>

That looks like this:

The truth is, I can’t be sure that you see this as I do. But I’ll assume that it looks idiotic, but OK. But what happens if a script puts more text than expected? Or if the user forces a larger font? I’ll try to simulate this by choosing a larger font here:

But the truth is that I can’t know if this example went through OK, because I don’t know how your browser responded. But that’s the point, isn’t it?

If you want to be on the safe side, use the ‘overflow’ property. Below is shown what happens when you set ‘overflow: hidden;’ (to the left) or ‘overflow: scroll;’ (to the right) for the toplevel <div>. At least nothing spills over.

Pitfall #6: Floats

For some reason, which is beyond me, floats are commonly suggested as a solution for organizing web pages (except for horizontal layout, as shown below). There is only one problem with floats: They were intended to push text, and nothing else.

The whole idea was to allow an image to float to the left or right, and let the text surround it, just like as they do in newspapers. This is why the CSS2 spec says, in paragraph 9.4.1:

In a block formatting context, boxes are laid out one after the other, vertically, beginning at the top of a containing block. (…) In a block formatting context, each box’s left outer edge touches the left edge of the containing block (for right-to-left formatting, right edges touch). This is true even in the presence of floats (although a box’s content area may shrink due to the floats).

Let me spell this out: Paragraphs, DIVs or whatever, shouldn’t get pushed aside by floats. What should get pushed is the text they contain (or more precisely, inline elements). What happens in reality seems to depend on the browser. Firefox and Google Chrome follow the spec, IE plays its own game.

So let’s look at the following example:

<div style="border: 1px solid black; margin: 6px; padding: 5px; float: left; width: 4em; height: 4em; background-color: #ffdddd; text-align: center;">
Nice image</div>
<div style="background-color: #ff8; width: 12em; border: solid 1px black;">
This is my DIV, which has melted into the float.
Only the text is pushed aside.
</div>

Which looks like this on your browser:

Note that the non-floating DIV (in yellow) should surround the float. It should be packed as if the float wasn’t there. This is what the spec says should happen. In real life I got:

Float behavior on Firefox and IE

Not very impressive, is it?

By the way, if you want a rectangle which doesn’t mix with floats, put a table there. Tables are inline elements, so they won’t collide with floats. But if you’re playing with floats, you must be thinking that tables are obsolete…

Alternatively, you could go for a floats-only region. People who test with IE only are likely to get this wrong, because of the problem mentioned above. Anyhow, the idea is to pack DIVs horizontally to the right and to the left using floats, and avoid collision from top and bottom using the “clear” directive. For example:

<div style="background-color: #ffd; border: solid 1px black; text-align: center;">Above all</div>
<div style="float: left; background-color: #fdd; width: 4em; height: 3em; border: solid 1px black; text-align: center;">
One</div>
<div style="float: left; background-color: #fdd; width: 4em; height: 2em; border: solid 1px black; text-align: center;">
Two</div>
<div style="float: right; background-color: #fdd; width: 4em; height: 4em; border: solid 1px black; text-align: center;">
Four</div>
<div style="float: right; background-color: #fdd; width: 4em; height: 3em; border: solid 1px black; text-align: center;">
Three</div>
<div style="clear: both; background-color: #ffd; border: solid 1px black; text-align: center;">And this is below all!</div>

Which looks like this:

This is great for multiple columns, whose heights are unknown. We want the bottom DIV below all of them, and this is how it’s done. Note that if we put a non-floating DIV in the middle, we would get different behavior on IE and W3C-compliant browsers.

Note that none of the DIVs in this example contains the other. What makes them avoid each other is the “clear: both” attribute at the bottom which tells the browser that floats are not allowed in either side. So we get a clean line, immediately below.

One problem (or feature) of this method, is that if there is not enough horizontal space, floats will be pushed vertically, a bit like inline flow. Paragraph 9.5 of the W3C specification says:

If there isn’t enough horizontal room on the current line for the float, it is shifted downward, line by line, until a line has room for it.

This can’t be relied on too much, though. It looks like Firefox prefers to let some of the page go out of view rather than to mess the page up, when the page is resized to very narrow (below 400 pixels?). I don’t know exactly how it works, but the fact is that at some point the floats don’t tile vertically (in IE they do). I guess it’s an interpretation of “not enough horizontal space”.

And just a final remark about floats: The W3C specification requires that the width is known, and must therefore be set unless it has an intrinsic value (e.g. an image). But nothing stops us from setting the width in percents. With this simple trick, we can have columns that grow and shrink according to the limiting DIVs width (resizing…), while controlling the proportions between the columns.

Just a warning, though: Be careful with letting the overall width come near 100%, since the browser may round each float’s pixel width slightly. And of course, keep in mind that the width doesn’t include the border, margins nor padding, so these can cause a mess when things get tight. Not to mention mixing percentage width floats with constant width floats.

Pitfall #7: Getting the selection wrong

The grammar is in principle simple, but knowing the formalities is important. For example, div.theclass { } means a DIV with class “theclass”, but not necessarily a DIV within it. For a DIV within a certain class, the selection is e.g. “.theclass DIV { }”. This also goes for IDs, e.g. “#theID DIV { }”. The list can be nested, so a link within a list item within a certain ID could look like “#theID li a:visited { }”. The same trick goes with classes instead of IDs.

If several cases are desired, start the nesting from the beginning. Exactly like “a.banner:link, a.banner:visited, a.banner:active { }” one should also go “banner a:link, banner a:visited, banner a:active { }” when a “within” relation is desired rather than a direct one.

Conclusion

If you want to control your page’s layout down to the pixel, DIVs and CSS are your screws and screwdriver. Just remember that things may look very different when someone views your page with a cellular phone or some other gadget you wouldn’t think about.

As they always say, when everything else fails, RTFM (Read the Fine Manual). In the case of CSS, that manual is the W3C specification. This is not to say that browsers really follow it. Firefox does most of the time. At least they consider it a bug when it doesn’t. Microsoft and their IE live in a world of their own.

Anyhow, if you wonder why your <div><div><div>-page is a mess, or why everything goes wrong with patches of page fragments being where they shouldn’t be, the ultimate answer is to understand how the fine machine works. Or at least, how it was meant to. And that is written in the spec. Boring, but a man has to do what a man has to do.

The “kamakesef” formula revealed: Calculate the wedding present’s amount

Introduction

An Israeli website, “Kama Kesef” (כמה כסף in Hebrew, meaning “how much money”) is a neat web calculator telling how much to give as a money present in weddings and other events. When I realized that I’m far from being the only one using it to decide what sum to write down on the cheque, I thought it’s time to investigate how the thing works. It turned out, that it’s quite simple, if you’re good in basic percent calculations.

Below is the procedure for making this calculation, as in March 2009. It’s expected to be updated from time to time, as prices change, of course. Using the website is easier, of course, but knowing what’s behind it can give an insight about Israeli social values. That said, one should remember that the calculator is offered by a wedding portal, and is therefore far from being a neutral source. After all, the wedding budget depends on the expectation from the money presents.

Overview

The procedure consists of seven stages. The first stage establishes the basic sum, and the other adjust it according to different criteria, as chosen by the user of the calculator. I will detail those stages in percents in most cases. If you want to follow the original procedure, don’t just sum up the percents. Rather, for each stage adjust the result from the previous stage. Also, note that each stage contributes a single factor or a single percent change, or no change at all. If more than one criterion matches, pick the one that matches best (for each stage, of course).

So ready? Let’s start.

Stage #1: What kind of event?

First we establish the initial amount. For a wedding, we start with 240 NIS. If it’s a Bar-Mitzva, Bat-Mitzva, Brit, “Brita” or a second wedding (!), it’s 160 NIS. A “hena” ceremony is 120 NIS. Birthdays, bachelor parties, bachelorette parties, and house-warming parties start this calculation with 100 NIS. And this is only the start…

In case you wonder, 240 NIS are grossly 60 USD or 40 Euro. When writing this, of course.

Stage #2: How many invited?

Most Israelis will not count the number invited, but the number arriving. Anyhow, if you come alone, stick to the initial amount. If you’re a couple plus children, double it. A couple alone should multiply by 1.8, but a parent and child should multiply by 1.3. Three adults (only) multiply by 2.5, four adults multiply by 3.2. If a child arrives with no parents, multiply by 0.87.

Stage #3: Relation

Being close to the married couple is expensive, we all know that. We start from the amount we got in the previous stage, and change as follows: If you’re the brother or sister, pump it up by 70%.  Nieces and nephews go up by 50%. Grandma and grandpa raise by 35%. Cousins add 23%, 2nd order cousins, aunts and uncles get away with 20% more. Distant relatives reduce by 10%, and the parents? They reduce 9%. After all, they pay for the wedding anyhow.

So much for the family. What about friends? If you’re THE closest friend, you’re like a brother, so you go 70% up. “Just close friends” raise by 50%. Distant friends, as well as sons and daughters of those being friends with the couple, don’t make any adjustment on this stage. Neighbours and coworkers reduce 5%.

Israel’s social culture in a nutshell.

Stage #4: Your occupation (or income)

At this stage, the user of the calculator picks either occupation or an income estimate. It’s not clear whether the latter is gross or net, but it looks like it’s net salary.

Anyhow, those who consider themselves plainly “employed” or “self-employed” jump to the next stage. The others take the amount they got in stage #3, and alter as follows:

If you’re a hi-tec worker, raise by 3%. Students and those “in financial difficulties” (?) allow themselves to drop 10%. Pensioners drop 11%, soldiers drop 20%, school children drop 25% and unemployed go down 30%. In the calculator you can only choose one category, so unemployed students in financial difficulties will have to do with only one discount.

Alternatively, you can pick your monthly salary: Those making 25000 NIS add 20%. If you get 16000 NIS, raise by 8%, for 12000 NIS raise by 5%. People making 8000 NIS will not change the amount on this stage (does it mean that the average employee in Israeli makes 8000 NIS?). Finally, the poor ones making 4000 NIS a month pay 15% less, and those making 2000 NIS a month (which is worse than being unemployed?) cut down 20%.

And a slightly bizarre option says, that if you’re Bill Gates, multiply by 6.

Stage #5: Where does the event take place?

If it’s in a hall, just go to the next stage. If it’s “the place”, raise by 5%. A garden or hotel will add just 1% to the sum. If it’s a Kibbutz, drop 5%. If it’s in a restaurant or more than 2 hours to drive, it’s 7% down. Those doing it in their own garden or home lose 10%. And if you can’t arrive, it’s 25% down. Now show me an Israeli who will send money to a wedding not attended.

What’s interesting about this stage, is that even though the calculator is hosted by a wedding portal (and hence with a clear interest) expensive places don’t get a significant boost here. This criterion takes the guest’s side, which is “if they picked a fancy place, why should we pay for it?” (My Polish genes stand out)

Stage #6: Winter or summer?

Almost done. We now take the amount found in stage #5, and give it a slight adjustment to compensate for the well-known fact, that weddings during the summer are more expensive. So if the wedding takes place during May to September, add some 10%, and also if it falls on Lag Ba’Omer (a Jewish holyday which is very popular for weddings), which is usually in this time span anyhow. April gets a raise by 5%, and March by 3%.

Otherwise, keep it as is.

Stage #7: Which weekday?

It’s well-known that weddings on Thursdays are best, because the guests can stay longer, as most of them don’t work on Friday. So Thursdays raise Stage #6′s amount by 10%. Tuesdays boost it by 5%. All other days, including these fancy Friday-noon weddings remain unchanged.

Conclusion

Despite the amount of details, it all boils down to a simple calculation. Beyond the immediate practical use of this formula (which is best exercised through the website) I think that this simple procedure is rare case, in which cultural values can be so directly evaluated with simple math.

And whoever or whatever it is you’re going to celebrate: Mazel Tov!

Using Perl to map FPGA pins from a board design to UCF pin constraints

One of the things I try to avoid as an FPGA engineer, is to manually configure the pin constraints (in the UCF file) in order to tell the tools which FPGA pin is connected to what. Not only is this extremely boring, but I also think that getting it done right (at the first go) is more or less a miracle.

If you insist on working with the Orcad schematics, you’re doomed. Yes, this graphical representation of the board is useful for getting an idea of what’s going on. But when I want to know for sure what is connected to what on the PCB, I read the netlist file. It’s that text file, which the board designer sends to the PCB manufacturing plant. So even if schematics is convincing, what counts is what the netlist file says. Sometimes reading the netlist reveals connections which were not obvious at all from looking at the schematics. But I’m diverting from the point, which is how to generate the UCF file sort-of automatically. Or at least spare most of the work.

So let’s have a look on what a netlist file looks like. This is a snippet from the middle, where the precious information is:

SIN_D8  = U43/V5 U48/42 ;
CLK_SEL  = R393/1 U52/36 ;
N14463023  = U29/7 R154/1 R326/2 ;
N16132557  = U64/1 C589/1 L90/1 L91/1
            C594/1 U64/2 ;
SOUT_A_A9  = U38/34 U43/AB21 ;
SENSOR_B_D3  = U43/N8 U49/11 ;
SIN_D9  = U43/U4 U48/44 ;

So the structure is very simple. A statement begins with the net’s name, an equation sign and then a listing of the connected pins. As you can see, each statement is terminated by a semicolon, and may consist of several lines.

The nets’ names are given by the board designer manually. There is no assurance that this name has anything to do with what the net is connected to, so if you’re really pedantic, you should verify that as well. Checking the schematics is fairly efficient, or you could verify the pin connections in the netlist, which may be fairly easy with some scripting skills.

Also, if no name was given to the net, but it’s a result of just connecting two pins, Orcad will make up a name, which usually looks something like N16132557. A common, and annoying case is when there is a resistor between two chips’ pins (say, for debouncing). Because of the resistor in the middle, two nets make the connection. If the board designer gives the name to the net between the FPGA and the resistor, we get the name for free. If not, we need to be smarter.

And that brings us to the pin listing  in the netlist. If we take the SOUT_A_A9 net for example, we can see that it’s connecting between pin 34 of device marked U38, and pin AB21 of U43. On this specific board, U43 happens to be the FPGA.

This leaves us with two possible strategies. One is to trust the net’s name, and make an entry in the UCF file, which binds pin AB21 to a Verilog/VHDL toplevel I/O port with a similar name, say “sout_a_a[9]“.  I’ll show an example script for this below.

The second strategy would try to find out what pin 34 of U38 stands for, and give the port’s name accordingly. This is trickier, of course, but given a reliable pin mapping of the other chip, this neutralizes any mistake possibly made by the board designer. This is where I’d like to mention, that certain board design tools create a “chip file”, which is a text file as well. This text file contains meaningful names for each chip included in the design (these are, in fact, the names that appear on the schematics). This file’s name is typically pstchip.dat. Unfortunately, the information in this file is commonly fed manually from a datasheet at some stage of the board design, so if an error was made during this stage, both the board and the FPGA pinout will get it wrong.

But let’s leave the pessimism for a while, and assume that we can rely on the nets’ names. Here’s a script, which finds the FPGA’s connections, an attempts to create a UCF file. Please keep in mind that it’s just an example, and that it doesn’t cover all nets even in the design I wrote it for. If you want to use this technique in your own designs, you’ll have to adapt it to the quirks of the netlist you’re facing.

Anyhow, here it is:

#!/usr/bin/perl
use warnings;
use strict;

undef $/; # Slurp mode. (Sane people use "local" instead)
my $file = <>;

my @chunks = ($file =~ /^([^ %].+?);/gsm);
my @out;

foreach my $chunk (@chunks) {
  my ($var, $rest) = ($chunk =~  /^([^ ]+)[ ]*=[ ]*(.*)/s);
  die("Failed to read line: $chunk\n")
    unless (defined $var);
  next if (grep { $_ eq lc $var }
	   qw[vcc_int vcc_aux v3_3 gnd]);
  my @pins = ($rest =~ /U43\/([^ \n\r\t]+)/gi);
  push @out, "NET \"".(lc $var)."\" LOC = \"$_\";\n" foreach (@pins);
}
print sort @out;

Just a few clarifications: U43 is the FPGA in my netlist, right? So that’s why I filter out anything else. And now a few Perl clarifications.

• I begin with undeffing $/. This makes the single ‘$file=<>’ statement read the entire file at once (that’s why they call it slurp mode). The Perl manpage encourages to use “local” instead of “undef”, and it explains why, but it’s not relevant for a short script.
• The first regular expression (feeding @chunks) cuts the file into pieces between semicolons.
• The second regular expression splits each chunk into the string before ‘=’ (in $var) and everything that comes afterwards (in $rest, possibly longer than a single line).
• The grep sentence checks if we’re not on a power net, which should be skipped.
• The last regular expression looks for a ‘U43/’-something, and if that is found, the pins are stored in the @pins list (in a sane case, this will be only one pin, or we’re messed up)

As I said before, I don’t really expect this script to work out of the box for you, but I hope I made the point about using a Perl script on a netlist to make life easier. And in case you’re an FPGA Engineer, and don’t know Perl, I hope this gave an idea of why you should start learning…

Encrypted disk, partition or USB stick on Linux: A short do-it-yourself cookbook

Before anything: Recent distros come with packaged utilities for encrypting a disk, partition, USB stick or whatever block device. This little cookbook is for those who are not that lucky, or prefer to do things with their bare hands. If you’re not familiar with using loop devices, I suggest playing a bit with them before going for the encrypted version. Because the whole idea behind the mechanism is to wrap the plaintext disk, partition or file with a loop device which does the encryption on the fly.

I’m running an old system, in terms of year 2009: A Red Hat 7.3 with kernel 2.4.21. This is why I was so surprised, when I got the encrypted loop device up and running in no time, despite the fact that it requires some patchwork. I repeated this process, with the same ease on a Fedora Core 9 machine with a 2.6-family kernel.

To be 100% fair, I’ll mention that both machines had the kernel sources properly installed, and with the sane options set. If you’re running on a home-compiled kernel, this will be your case as well.

So here is the quick summary of actions:

1. Downloaded util-linux-2.12r and loop-AES-v3.2e. Opened tarballs.
2. Patched the util-linux to support loop-AES. At util-linux-2.12r’s root directory, I went:

   patch -p1 -i ../loop-AES-v3.2e/util-linux-2.12r.diff

   This makes losetup, mount and friends to respect the necessary flags. Note that the patch for util-linux is given by loop-AES. This is why downloading two recent versions is a good idea (or the two I mentioned).

3. Compilation. At the same directory:

   CFLAGS=-O2 ./configure
   make SUBDIRS="lib mount"

The configuration phase went with some scary messages, but they turned out irrelevant. At this point, I had the relevant utilities as executables in the “mount” subdirectory

If you’d like to compile these as static binaries, this should be typed before anything:

export LDFLAGS="-static"

4. Next step was to compile the loop kernel module. The whole idea is to replace the existing module with one that supports AES encryption.  It means that the kernel is patched, but only the specific module is compiled and replaced. This is a pretty scary approach, but it works. So At loop-AES’ home directory, just go, as root:

   make clean
   make

   I know. I never compile as root. But this one does the work too well. Note that there is no “make install” because plain “make” includes the installation part.

5. Next I installed the utilities. But I didn’t want to overwrite the original binaries. Why change the existing mount and swapon on a working system? So the idea is to put the executables close to the original ones, only with a “-patched” suffix.

   So from the util-linux-2.12r/mount/, I ran as root:

   for i in losetup mount pivot_root swapon umount ; do cp $i `which $i`-patched ; done

6. And a final test: (my USB stick encrypted)

   losetup-patched -p 0 -e AES128 /dev/loop0 /dev/sdb1 < password

   Ah, before this, I prepared a file called “password”, containing the secret which is necessary to open the encrypted block device. Without the “-p 0″ flag and injecting the password through standard input, the user has to type it manually. Since at least 20 characters are required, the common solution is to let the user type a shorter password, and scramble it into something longer. Of course it’s completely stupid to keep the password on the disk as plain text. You might as well not encrypt anything if you do this.

7. Played around with the new loop device, formatted it, mounted it with the old “mount”, unmounted, and finally released the loop device (with old losetup)

   losetup -d /dev/loop0

That’s it. May look a bit scary, but it was really easy.

Why MySQL’s (SQL) DATETIME can and should be avoided

SQL DATETIME sucks

MySQL, among other databases, has a column type called DATETIME. Its name seems to mislead people into thinking that it’s suitable for storing time of events. Or suitable for anything.

This is a general SQL thing, by the way, but I’ll demonstrate it on MySQL.

I often find this column type in other people’s database schemas, and I wonder if the designer gave it a thought before using it. It’s true, that in the beginning it looks simple:

mysql> CREATE TABLE stupid_date ( thedate DATETIME, PRIMARY KEY (thedate) );
Query OK, 0 rows affected (0.04 sec)

mysql> INSERT INTO stupid_date(thedate) VALUES ( NOW() );
Query OK, 1 row affected (0.03 sec)

mysql> SELECT * FROM stupid_date;
+---------------------+
| thedate             |
+---------------------+
| 2009-03-15 14:01:43 |
+---------------------+
1 row in set (0.00 sec)

That was way too cute, wasn’t it? We also have the NOW() function, which fits in exactly, and puts the current time! Yay! Yay! And if the timestamp looks old-fashioned to you, I suppose there is a reason for that.

But wait, there are two major problems. The first one is that the time is given in the host’s local time. That was fair enough before the internet was invented. But today a web server can be a continent away. DATETIME will show you the local time of the server, not yours. There are SQL functions to convert timezones, of course. Are you sure that you want to deal with them? What happens when you want to move your database to a server in another timezone? What about daylight saving time? Local time is one big YUCK.

(Update: As commented below, the real stupidity is to use NOW(), and not UTC_TIMESTAMP(). The latter gives the UTC time, as its name implies)

Problem number two: Most applications don’t care what the absolute time is. The current time is commonly used to calculate how much time has elapsed since a certain event. To filter elements according to if they were X seconds before now. Is the user still logged in? Has 24 hours elapsed since the last warning email was sent? And so on.

“Solution”: The SQL language supplies a variety of COBOL-like functions to calculate whatever we can ask for. And also an opportunity to get things horribly wrong, because the SQL statement became way too complicated.

Use POSIX time() instead

Sorry, didn’t mean to scare you off. It’s really simple: Any modern operating system, even Windows, will readily supply you with the number of seconds since January 1, 1970, midnight, UTC (that is, more or less GMT). This is also called “seconds since the Epoch” or “UNIX time”.

No matter where the computer is, what timezone it uses or what programming language you’re using, this simple integer representation will show the same number at any given moment.

You can, in fact, obtain this number from MySQL directly:

mysql> SELECT UNIX_TIMESTAMP(thedate) FROM stupid_date;
+-------------------------+
| UNIX_TIMESTAMP(thedate) |
+-------------------------+
|              1237118503 |
+-------------------------+
1 row in set (0.00 sec)

This means, that 1237118503 seconds elapsed since the Epoch (which is a global time point) until 14:01:43 in Israeli LOCAL time of the day I wrote this post. So now we have an integer number to work with, which is handy for calculations, but things will still get messy if we try to move the database to another server.

Store the number instead

If we are interested in working with integers, why not store the integer itself in the database? We could go:

mysql> CREATE TABLE smart_date ( thedate INTEGER UNSIGNED, PRIMARY KEY (thedate) );
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO smart_date(thedate) VALUES (1237118503);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM smart_date;
+------------+
| thedate    |
+------------+
| 1237118503 |
+------------+
1 row in set (0.00 sec)

That wasn’t very impressive, was it? The first question would be “OK, how do I get this magic number, now that I don’t have the NOW() function?”

The short and not-so-clever answer is that you could always use MySQL’s UNIX_TIMESTAMP( NOW() ) for this. The better answer is that no matter which scripting or programming language you’re using, this number is very easy to obtain. I’ll show examples below.

As for the magnitude of this number, yes, it’s pretty big. But it will fit a signed 32-bit integer until year 2038. I presume that nobody will use 32-bit integers by then.

And finally, one could argue that DATETIME is convenient when reading from the database directly. True. But for that specific issue we have the FROM_UNIXTIME() function:

mysql> SELECT FROM_UNIXTIME(thedate) FROM smart_date;
+------------------------+
| FROM_UNIXTIME(thedate) |
+------------------------+
| 2009-03-15 14:01:43    |
+------------------------+
1 row in set (0.00 sec)

And again, this is given in the computer’s local time. Which is fine, because it’s intended to be read by humans. In particular, humans who easily translate time differences between their server and themselves.

Obtaining Epoch time

Just to prove that it’s easy to know what the “Epoch time” is in any language, here are a few examples. Wherever it’s really simple, I’m showing how to convert this format to human-readable format.

In Perl:

print time();
print scalar localtime time(); # Local time for humans

In PHP:

<?php
echo time();
echo date('r', time() ); // Local time for humans
?>

In Python:

from time import time;
print time();

(note that the time is returned as a float number with higher precision)

In C:

#include <time.h>
#include <stdio.h>

int main () {
  int now = time(NULL);

  printf("%d seconds since the Epoch\n", now);
  return 0;
}

In JavaScript:

<script language="JavaScript" type="text/javascript">
now = new Date();
alert( now.getTime() / 1000 );
</script>

In this case, the time is shown with a fractional resolution.

The JavaScript example is not really useful for a database application, because the time is measured at the computer showing the page. In a website application, this is just anybody’s computer clock, which may be wrong. But it’s yet another example of how this time representation is widely available.

Conclusion

Drop those DATETIME columns from your tables, and use a simple, robust and handy format to represent time. Don’t let the database play around with a sensitive issue like time, and don’t risk getting confused by different functions when calculating time differences. Just because the DATETIME column type exists, it doesn’t mean there is a reason to use it.

Enjoy the database on what it’s best at: Storing and collecting information.

Xilinx’ XST synthesizer bug: ROM generation using case

Take a close look on the Verilog code below. This is a plainly-written synchronous ROM. Do you see anything wrong with it? (Spoiler: There is nothing wrong with it. Not that I know of)

module coeffs
  (
   clk, en,
   addr, data
   );

   input clk, en;
   input [9:0] addr;
   output [15:0] data;

   reg [15:0]      data;

   always @(posedge clk)
     if (en)
       case (addr)
     0: data <= 16'h101a;
     1: data <= 16'h115b;
     2: data <= 16'h0f1c;
     3: data <= 16'h0f6d;
     4: data <= 16'hffa4;

... and counting up ...

     249: data <= 16'h0031;
     250: data <= 16'hfffa;
     251: data <= 16'hffee;
     default: data <= 0;
   endcase
endmodule

But it so happens, that Xilinx’ XST synthesizer failed to get this one right. XST J.39, release 9.2.03i, if you insist.

And when I say it didn’t get it right, I mean that what I got on the hardware didn’t implement what the Verilog says it should.

First, what it should have done: Since the address space consists of 10 bits, and there are a lot of, but less than 1024 data elements, the synthesizer should have matched this with a 1k x 18 block RAM, set the values as INIT parameters, and not allow any writes. And so it did. Almost.

The problem, it seems, lies in the fact that only 252 data slots are assigned, leaving 3/4 of the ROM with zeroes. This is where the synthesizer tried to be smarter, for no practical reason. Based upon what I saw with the FPGA Editor, the synthesizer detected, that if any of addr[9] or addr[8] are nonzero, then the output is zero anyhow. Since the block RAM has a synchronous reset input, which affects only the output, the synthesizer decided to feed this reset with (addr[9] || addr[8]). This doesn’t change anything: If any of these lines is high, the output should be zero. It would be anyhow, since the block RAM itself contains zeros on the relevant addresses, but this reset logic doesn’t hurt. As long as you get it right, that is. Which wasn’t the case this time.

What really happened, was that the synthesizer mistakenly reversed the polarity of the logic of the reset line, so it got (!addr[8] && !addr[9]) instead. That made the memory array produce zeros for any address. And the design didn’t work.

It looks like the idea was to reverse the polarity at the block RAM’s reset input as well (which costs nothing in terms of logic resources) but somehow this didn’t come about.

Workaround: It looks like the “default” statement triggered this bug. Since the Verilog file was generated by a computer program anyhow, I let it go on running all the way to 1023, explicitly assigning zeros to each address. This is completely equivalent, of course, but made the design work in real life.

One of these bugs you wouldn’t expect.

BLOB, TEXT, and case sensitivity: MySQL won’t treat them the same

When I first discovered that there is both BLOB and TEXT in databases, I was puzzled. They occupy the same amount of disk space, and the database doesn’t alter the data itself anyhow. Why two?

Of course, I followed the stream, and went for TEXT and VARCHAR for everything, since I don’t store binary data in the database. That may not be the optimal choice in all cases.

It turns out, that MySQL goes a long way to “help” the user with string operations. In particular, when the table column is defined as TEXT, VARCHAR and their derivatives, the database will compare strings as they would be understood by humans. And if that definition sounds ambiguous to you, you’re in good company: Different versions of MySQL compare text strings differently. For example, snipping leading and trailing whitespaces from the strings before comparing them: Some versions will do this, others won’t.

The bottom line is that if you change your MySQL server, tiny bugs may creep in. All these corner cases may behave differently. This is the classic case of some entry disappearing from a list of 53478, without anyone noticing.

Another issue to consider is character collation: When using TEXT, VARCHAR and friends, we also have that ‘Ö’ and ‘OE’ are treated as the same character (when German character set is used, among others). Just an example.

The solution: Make MySQL treat your data as a binary. A BLOB column type, for example. You loose all those extra “features”, but gain stability over database versions and flavors. That means that you have to handle all the case-insensitivity issues yourself, as well cleaning up the strings properly. With good practices, that’s not an issue. It’s a matter of if you want to take responsibility, or let the database iron those small wrinkles for you.

And finally: What about uniqueness in tables? Here’s a short session, using MySQL 4.0.24:

mysql> CREATE TABLE try (mydata TEXT, UNIQUE INDEX (mydata(20)) );
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO try(mydata) VALUES('Hello');
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM try WHERE mydata='hELLO';
+--------+
| mydata |
+--------+
| Hello  |
+--------+
1 row in set (0.00 sec)

This was pretty much expected: As a TEXT column, the comparison was case-insensitive. And of course, the capital “H” was saved in the table, even though that doesn’t matter in string comparisons.

But what happens if we want to add an entry, which violates the uniqueness, when considering the strings in a case-insensitive manner?

mysql> INSERT INTO try(mydata) VALUES('HELLO');
ERROR 1062: Duplicate entry 'HELLO' for key 1

As expected, MySQL didn’t swallow this. “Hello” and “HELLO” are the same, so they can’t live together when “mydata” is restricted as UNIQUE.

So let’s drop the table, and try this again, this time with a BLOB column. Spoiler: Everything is case-sensitive now.

mysql> DROP TABLE try;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE try (mydata BLOB, UNIQUE INDEX (mydata(20)) );
Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO try(mydata) VALUES('Hello');
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM try WHERE mydata='hELLO';
Empty set (0.00 sec)

(why should it find anything? ‘Hello’ and ‘hELLO’ are completely different!)

mysql> SELECT * FROM try;
+--------+
| mydata |
+--------+
| Hello  |
+--------+
1 row in set (0.00 sec)

mysql> INSERT INTO try(mydata) VALUES('HELLO');
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM try;
+--------+
| mydata |
+--------+
| Hello  |
| HELLO  |
+--------+
2 rows in set (0.01 sec)

(No problems with the uniqueness: ‘Hello’ and ‘HELLO’ are not the same in a BLOB)

To summarize all this: Before choosing between TEXT or BLOB, ask yourself if you want the database to treat the string exactly as it is, or if you want some forgiveness regarding case, whitespaces and natural language issues.

For example, are you sure that you want the user name and password as text? In particular, would you like the password case-insensitive? Do you want HTTP links as text? The address itself is indeed case-insensitive to the web, but CGI arguments (everything after the question mark, if present) is case-sensitive (YouTube video IDs, for example).

Usually, using a text column is OK. But it’s a choice one has to make.

Using ImageMagick to convert a 4:2:2 YCrCb raw image to something viewable

Some electronic imaging system dumped a 4:2:2 YCbCr raw image into a file. It’s a debug output. Now I wanted to see this image. GIMP doesn’t import that format. ImageMagick had the solution. It was as simple as

convert -size 800x600 pal:ImageOut.raw ImageOut.bmp

I don’t know why they picked the codename “pal” to represent 4:2:2 format (maybe because of ITU-R BT.601?) , but it did the job. Ah, they call it “YUV” and not “YCbCr”, but it was really close enough for me.

For a list of supported formats:

convert -list format

Xilinx FPGA MPPR script

I reached that point, at which place and route sometimes met timing, and sometimes it didn’t. It was all a matter of playing with the placer cost table number. The FPGA guy’s gambling for lucky numbers.

The ISE tool (Xilinx’ native IDE) supports an feature called Multi-Phase Place and Route (MPPR for short), which basically means that the tools will run several phases of place and route, with different placer cost table figures. I never tried it, to be honest. Since I’m not very fond of IDE’s in general, and not using ISE in particular, I preferred to do it myself.

Besides, MPPR will run for a given number of times. I want the computer to loop until the constraints are met. No more, no less.

So here is the script I used. If you’re using Linux or Cygwin, it will be pretty easy to modify it for your uses.

#!/bin/bash

projectname=myproj
reportfile=$projectname.par
cost=1

while (( cost<100 )) && [ -e $reportfile ] && ! grep -q 'All constraints were met' $reportfile ; do
  rm -f $reportfile;
  C:\\WINNT\\system32\\cmd.exe /c start "" /b /low /wait par -intstyle ise -w -ol med -pl med -rl med -t $cost "$projectname"_map.ncd $projectname.ncd $projectname.pcf
  if ! grep -q 'All signals are completely routed' $reportfile ; then
    echo PAR seems to have failed. Stopping.
    exit 1;
  fi
  echo Done PAR with cost table $((cost++))
done;

echo Done.

Now, if you’re bothered by the line going

C:\\WINNT\\system32\\cmd.exe /c start "" /b /low /wait par -intstyle ise (...)

you may exchange it with simply

par -intstyle ise -w -ol med -pl med -rl med -t $cost "$projectname"_map.ncd $projectname.ncd $projectname.pcf

The only reason I use cmd.exe with all its parameters, is that I want the PAR process to run with a low priority. After all, I may want to do something else with my computer while this CPU hog is running.