my tech blog

Avoiding reboot: Resetting USB on a Linux machine

Every now and then, some USB device misbehaves badly enough to knock out the entire interface, to the extent that the system doesn’t detect any new USB devices. Or work so well with the existing ones, for that matter. The solution for me until now was to reboot the computer. But hey, I don’t like rebooting Linux!

So this script (I call it usbreset) performs a reset to the USB drivers as necessary to bring them back to life. As a side effect, an equivalent to unplugging and replugging all USB devices takes place. If possible, unmount any connected USB storage device (e.g. disk on key) before doing this. Also, if you’re in the middle of printing through a USB device, or some other operation, this will not be graceful.

An interesting thing I noted, was that if there’s a really problematic device on the bus (for example, a device that fails to enumerate), this script doesn’t return. It probably gets stuck on one of the writes to “bind” (hasn’t checked that all the way through).

Use at your own risk. You are root, you should know what you’re doing (or gamble like myself). Script follows.

If it doesn’t work for you, check the comments below. The names of the /sys files vary slightly among different distributions.

A word about how this script works, so it goes like this: The USB controllers are PCI / PCIe / fake PCI devices. So the script goes to the /sys directory that is related to the PCI devices’ driver, and unbinds them from the device, and then bind them back. So it’s like saying, nah, this device doesn’t belong to the driver, and then say, well, it is. And in the latter stage, the driver initializes the device, which does the job. On a good day.

Those file names like 0000:00:14.0 that appear in /sys/bus/pci/drivers/xhci_hcd/ are the PCI bus addresses of the USB controllers that are handled by the xhci_hcd driver. These will vanish from the directory after writing to “unbind” and return after writing to “bind”. The names can be found in /sys/bus/pci/devices/. Use “lspci” to tell which device is which.

#!/bin/bash

if [[ $EUID != 0 ]] ; then
  echo This must be run as root!
  exit 1
fi

for xhci in /sys/bus/pci/drivers/?hci_hcd ; do

  if ! cd $xhci ; then
    echo Weird error. Failed to change directory to $xhci
    exit 1
  fi

  echo Resetting devices from $xhci...

  for i in ????:??:??.? ; do
    echo -n "$i" > unbind
    echo -n "$i" > bind
  done
done

The script above is in fact a newer version of the script, which I updated to in June 2014, after upgrading my kernel to 3.12. The older one, to which some of the comments relate, is this one:

#!/bin/bash

SYSEHCI=/sys/bus/pci/drivers/ehci_hcd
SYSUHCI=/sys/bus/pci/drivers/uhci_hcd

if [[ $EUID != 0 ]] ; then
 echo This must be run as root!
 exit 1
fi

if ! cd $SYSUHCI ; then
 echo Weird error. Failed to change directory to $SYSUHCI
 exit 1
fi

for i in ????:??:??.? ; do
 echo -n "$i" > unbind
 echo -n "$i" > bind
done

if ! cd $SYSEHCI ; then
 echo Weird error. Failed to change directory to $SYSEHCI
 exit 1
fi

for i in ????:??:??.? ; do
 echo -n "$i" > unbind
 echo -n "$i" > bind
done

The problem with the old script is that the EHCI subdirectory doesn’t appear on my system, because xHCI is a catch-all for all USB versions, hence when a USB port is 3.0 capable, it’s driven by the xhci_hcd driver only. The new script catches all possibilities. Hopefully.

Mini DV MD80 thumb camera: Usage notes with Fedora 12

So I bought this $10 mini DV camera on EBay. Since the user’s manual is pretty out of sync with itself and the product (Chinese and English doesn’t match, even by looking at the specification and drawings) I’ve written down some basic howto, in case I need to remember what to do at a later stage.

I can’t say I’m disappointed, because I didn’t really expect things to work as specified on a $10 camera.

Even though the manual promises 70 minutes of recording time, I got 48 minutes in reality before the camera stopped itself (the memory card wasn’t full, it was the battery that ran out).

Note that a MicroSD card needs to be purchased separately, which will cost significantly relative to the camera’s price (some $8 in a store, possibly $4 on EBay for a Sandisk card). The largest file I managed to obtain with a 48 minutes recording was 1.7 GB, so a 2 GB card is probably large enough.

LED colors

It looks like the LED colors were changed since the manual was written. The blue LED is the power LED, indicating that the device is on. The red LED says something is happening.

Optics and image quality

The image is a 720x480, apparently progressive (non-interlaced) rolling shutter with a noise level that resembles web cameras from the late 90′s. The measured opening angle on the frame’s width was as narrow as 25 degrees which, I suppose is about 80mm focal length on a 35mm-equivalent scale. In other words, this is with a slight touch of a tele lens.

The camera was announced having an 80 degrees view angle by its EBay seller, which turned out to be wrong. The narrow angle is a problem for most relevant applications, since it the desired subject gets off-frame easily. It’s also the reason for the shaky footage this camera emits. I’m not even sure about using this as a helmet camera.

With a $1 Jelly Lens, the angle of view rises to 40 degrees, which is around 50mm focal length (on a 35mm scale). In other words, the extra lens is some x0.6 (not very impressive, but what did I expect from a $1 lens?) and it gives the camera a “normal” focal length. The sticky adhesive on the jelly lens held the very small piece of plastic firmly in place. I don’t know how well this would work on a helmet cam, though. Did I say $1?

I should mention, that for a cellular phone with an already pretty wide angle, this Jelly Lens actually achieves an impressive wide angle (and green visible borders). So it’s a very good deal, given its price…

Another lens tried out was the AGPtek “180 Degrees” lens for cellular phones for some $5. It is often announced as x0.28, but it’s not. My measurement was 53 degrees, which is about 36 mm focal length (on a 35mm scale), so the lens did in fact x0.45. Better than the Jelly Lens, but by far not as good as expected. It does have a slight roundoff in the corners like a fish-eye lens at 53 degrees view angle, which gives the illusion that it’s wider than it actually is.

Charging

Connect the camera to a computer via USB, so it gets power. The blue LED goes on, and the red LED starts blinking. According to the user’s manual, a green LED should be blinking, but it looks like the color was changed to red. The user’s manual also says that the LED should stop blinking when the battery is charged, but that didn’t happen even after an overnight charging. It just went on blinking (red). According to the seller at EBay, the charging time should be 3 hours. Go figure.

Note that the MicroSD is mounted on the computer due to the USB connection.

Accessing the MicroSD card

Well, simply connect to the computer (as in for charging). The following (or similar) will appear at the log:

Jan 31 17:54:27 myhost kernel: hub 1-2:1.0: unable to enumerate USB device on port 2
Jan 31 17:54:29 myhost kernel: usb 1-2.2: new high speed USB device using ehci_hcd and address 62
Jan 31 17:54:29 myhost kernel: usb 1-2.2: New USB device found, idVendor=04d6, idProduct=e101
Jan 31 17:54:29 myhost kernel: usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 31 17:54:29 myhost kernel: usb 1-2.2: Product: usbdisk  
Jan 31 17:54:29 myhost kernel: usb 1-2.2: Manufacturer: anyka    
Jan 31 17:54:29 myhost kernel: usb 1-2.2: SerialNumber: 942954944
Jan 31 17:54:29 myhost kernel: scsi102 : usb-storage 1-2.2:1.0
Jan 31 17:54:30 myhost kernel: scsi 102:0:0:0: Direct-Access     anyka    MMC Disk         1.00 PQ: 0 ANSI: 2
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: Attached scsi generic sg4 type 0
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] 3858560 512-byte logical blocks: (1.97 GB/1.83 GiB)
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] Write Protect is off
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] Assuming drive cache: write through
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] Assuming drive cache: write through
Jan 31 17:54:30 myhost kernel: sdd:
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] Assuming drive cache: write through
Jan 31 17:54:30 myhost kernel: sd 102:0:0:0: [sdd] Attached SCSI removable disk

This is just like connecting a disk-on-key, and so is the access to the content.

Using as a webcam

This is useful for getting an idea of the image quality, and immediate feedback for aiming the camera properly.

With the camera connected via USB (and hence the MicroSD card mounted), unmount the volume (“Safely Remove Drive” or something) and press the camera’s Power button for three seconds or so. The red LED will stop blinking, and the blue remains steadily on. The following lines in the log indicate the transformation into a web cam:

Jan 31 17:55:20 myhost kernel: usb 1-2.2: USB disconnect, address 62
Jan 31 17:55:21 myhost gnome-keyring-daemon[28416]: removing removable location: /media/New flash
Jan 31 17:55:21 myhost gnome-keyring-daemon[28416]: no volume registered at: /media/New flash
Jan 31 17:55:21 myhost gnome-keyring-daemon[3139]: removing removable location: /media/New flash
Jan 31 17:55:21 myhost gnome-keyring-daemon[3139]: no volume registered at: /media/New flash
Jan 31 17:55:23 myhost kernel: usb 1-2.2: new high speed USB device using ehci_hcd and address 63
Jan 31 17:55:23 myhost kernel: usb 1-2.2: New USB device found, idVendor=04d6, idProduct=e102
Jan 31 17:55:23 myhost kernel: usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 31 17:55:23 myhost kernel: usb 1-2.2: Product: UVC..
Jan 31 17:55:23 myhost kernel: usb 1-2.2: Manufacturer: ANYKA
Jan 31 17:55:23 myhost kernel: usb 1-2.2: SerialNumber: 12345
Jan 31 17:55:23 myhost kernel: uvcvideo: Found UVC 1.00 device UVC.. (04d6:e102)
Jan 31 17:55:23 myhost kernel: uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround.
Jan 31 17:55:23 myhost kernel: input: UVC.. as /devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.2/1-2.2:1.0/input/input13

The last row indicates the generation of /dev/video0:

$ ls -l /dev/video0
crw-rw----+ 1 root video 81, 0 2013-01-31 17:55 /dev/video0

It’s important to start doing something with the webcam soon, or the camera goes into sleep mode. Running Cheese Webcam Booth (2.28.1) shows an immediate image. No configuration should be necessary (it should find /dev/video0 automatically).

Using as a stand-alone camera

According to the manual, the camera has two modes: Normal recording and audio-triggered. I haven’t tried the audio-triggered mode, and neither do I want to. To switch from one mode to another, there’s the “Mode” button. Which I’m not touching.

Needless to say, a MicroSD card must be inserted for this to work.

To start, make sure that the device is disconnected from the computer and off (no LED is on). Press the Power button for a second, wait a few seconds. Only the blue LED should lit steadily.

To start and stop recording, press the button on the camera’s short edge (next to the record/stop symbols). The red LED will blink at 0.5 Hz during recording.

If the “Mode” button is pressed, the red LED will blink at 2-3 Hz to indicate audio-triggered recording. Turning the camera off and on is the best way to make sure the camera is at a known state.

Playback

The video files are put in the “VIDEO” subdirectory. A typical playback session with mplayer looks like this:

$ mplayer "/media/New Flash/VIDEO/2012-9-21 18-26-54.AVI"
MPlayer SVN-r31628-4.4.4 (C) 2000-2010 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing /media/New Flash/VIDEO/2012-9-21 18-26-54.AVI.
AVI file format detected.
[aviheader] Video stream found, -vid 0
[aviheader] Audio stream found, -aid 1
VIDEO:  [MJPG]  720x480  24bpp  30.000 fps  2449.9 kbps (299.1 kbyte/s)
Clip info:
 Software: ankarec
Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory
[vdpau] Error when calling vdp_device_create_x11: 1
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffmjpeg] vfm: ffmpeg (FFmpeg MJPEG)
==========================================================================
==========================================================================
Opening audio decoder: [pcm] Uncompressed PCM audio decoder
AUDIO: 8000 Hz, 1 ch, s16le, 128.0 kbit/100.00% (ratio: 16000->16000)
Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)
==========================================================================
AO: [pulse] 8000Hz 1ch s16le (2 bytes per sample)
Starting playback...
Movie-Aspect is undefined - no prescaling applied.
VO: [xv] 720x480 => 720x480 Planar YV12
A:   3.0 V:   3.0 A-V:  0.001 ct:  0.002  92/ 92  5%  0%  0.1% 0 0

The video format is MJPEG 720x480 (not sure about the aspect ratio) with uncompressed 16 bit per sample mono sound, at 8 kHz sample rate. The file name represents the time at which recording began.

A small timestamp appears at the lower right of the recorded image.

Setting the time

This is somewhat confusing. The way to set the time, is to create a file called time.txt in the card’s root directory. To make things a bit complicated, the camera creates a file called TIME.TXT in the same place, with a timestamp sample (or something). Editing this file will do nothing. It’s a new file that needs to be created with something like

$ vi "/media/New Flash/time.txt"

It looks like the file appears from nowhere, with a sample timestamp (2012-09-21 17:08:56) set. Update it to something like

2013-01-31 19:45:00

or write it from scratch if nothing appears. Write and quit vi. Unmount the volume, unplug the camera, and turn it off. The camera will update the time to the file’s content when it starts up the next time.

There’s a null-character in the end in the sample timestamp. It can be omitted or left. Doesn’t matter.

Conclusion

Pretty much expected, it’s a piece of junk with poor documentation. Otherwise, it wouldn’t go for $10. But it’s good for putting in places where it has a good chance to get lost or destroyed, and hopefully get some cool footage when put on a helmet or something like that.

Xilinx “map” tool trimming just a little too much

Sometimes, in particular when working on a relatively new Xilinx device family, the “map” tool fails on several errors like

ERROR:MapLib:979 - LUT5 symbol
"project/project_core/module/module_empty_user_w_smb_wren_d_O
R_201_o_inv1" (output
signal=project/project_core/module/module_empty_user_w_smb_wr
en_d_OR_201_o_inv) has input signal
"project/project_core/module/module_almostfull_d" which will
be trimmed. See Section 5 of the Map Report File for details about why the
input signal will become undriven

The first thing to say about this, is that it’s a bug in Xilinx’ tools. Sooner or later, Xilinx will announce that it has been fixed, and ask people to upgrade to recent versions (like in this case).

What the message says is more or less

Hi, this is the mapper. I was too eager to throw away logic which I thought was useless, but oops, it turns out I need this piece of logic after all. Please look in Section 5, and try to figure out why I made this mistake.

My own experience with solving this problem is that there is no quick fix. No flag to add or anything like that. The trick is to find what provoked the mapper into removing logic eagerly (usually whole blocks of logic) and accidentally remove a few elements too much. Look in section 5 for the signal that was trimmed (called “input signal” in the error message), and try to spot what triggered off the logic removal. It’s usually a significant chunk of logic (a functional block) which is effectively useless, because it’s fed with constant signals. A very possible reason for this is that those signals aren’t even connected in the module’s instantiation, so they are all held constant zero. So the mapper rightfully removes this dead weight. Just a bit too much.

Another thing to mention in this context, is that this problem always occurs with black-box logic in the design, that is, IP cores (e.g. FIFOs) and other logic blocks that are included as netlists to the project (as opposed to HDL sources). Dead logic is optimized out by the synthesizer when possible, so the only reason for dead logic in the mapping stage is that it came from different netlists.

So the exact thing to look for is an IP core or another black box that was instantiated with some of its ports either constant or ignored, causing significant parts of it turning into dead logic.

And finally, let’s say this again: This is a bug in Xilinx’ tools, and one of the reasons why working with new device families is a bit difficult. There is no excuse for failing a build because some of the logic is unused.

A perl script sending mails for testing a mail server

Just set up your mail server? Congratulations! Now you should test it. In particular, check if it relays mails to other servers and if the response time is reasonable. Here’s a script for doing the testing. Just edit the arguments to send_mail() to match your setting.

#!/usr/bin/perl
use warnings;
use strict;
use Net::SMTP;

send_mail('127.0.0.1', # Host
 'sender@nowhere.com', #From
 'myself@myhost.com', #to
 'Just a test, please ignore',  #Message body
 "Testing email.\n" # Subject
 );

sub send_mail {
 my ($SMTP_HOST, $from, $to_addr, $body, $subject, $msg) = @_;

 $msg = "MIME-Version: 1.0\n"
 . "From: $from\n"
 . "To: " . ( ref($to_addr) ? join(';', @$to_addr) : $to_addr ) . "\n"
 . "Subject: $subject\n\n"  # Double \n
 . $body;

 #
 # Open a SMTP session
 #
 my $smtp = Net::SMTP->new( $SMTP_HOST,
 'Debug' => 1,       # Change to a 1 to turn on debug messages
 Port => 587,
 );

 die("SMTP ERROR: Unable to open smtp session.\n")
 if(!defined($smtp) || !($smtp));

 die("Failed to set FROM address\n")
 if (! ($smtp->mail( $from ) ) );

 die("Failed to set receipient\n")
 if (! ($smtp->recipient( ( ref($to_addr) ? @$to_addr : $to_addr ) ) ) );

 $smtp->data( $msg );

 $smtp->quit;
}

Two things to note:

The Port assignment marked red above makes an encryption connection with the server. It can be changed to 25, but many servers don’t answer strangers on that port.

July 2024 update: Nowadays, the situation seems to be the opposite. Mail servers that appear in the MX records and owned by Internet actors seem to answer only to port 25, and apparently only small servers answer to 587. Possibly because port 587 is commonly used for outgoing mail by MUAs (mail clients wishing to send an email) and port 25 is used only between servers (MTAs) …?

And if this script is used to talk with a remote server, odds are it won’t work due to authentication issues. If your server runs sendmail, it can be made less picky by making the following temporary changes to allow for testing:

In /etc/mail/sendmail.mc, change confAUTH_OPTIONS from `A’ to `’ (nothing, no authentication required). Also, change

DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl

to

DAEMON_OPTIONS(`Port=submission, Name=MSA')dnl

and then compile the configuration file and restart the server with

# make -C /etc/mail
# service sendmail restart

Needless to say, it’s recommended to return the original settings after the testing is done. Your mail server should have some self-respect.

Anyhow, a typical output should look like this:

Net::SMTP>>> Net::SMTP(2.31)
Net::SMTP>>>   Net::Cmd(2.29)
Net::SMTP>>>     Exporter(5.62)
Net::SMTP>>>   IO::Socket::INET(1.31)
Net::SMTP>>>     IO::Socket(1.30_01)
Net::SMTP>>>       IO::Handle(1.27)
Net::SMTP=GLOB(0x7c9d98)<<< 220 myhost.localdomain ESMTP Sendmail 8.14.4/8.14.4; Mon, 14 Jan 2013 14:03:26 +0200
Net::SMTP=GLOB(0x7c9d98)>>> EHLO localhost.localdomain
Net::SMTP=GLOB(0x7c9d98)<<< 250-myhost.localdomain Hello localhost.localdomain [127.0.0.1], pleased to meet you
Net::SMTP=GLOB(0x7c9d98)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP=GLOB(0x7c9d98)<<< 250-PIPELINING
Net::SMTP=GLOB(0x7c9d98)<<< 250-8BITMIME
Net::SMTP=GLOB(0x7c9d98)<<< 250-SIZE
Net::SMTP=GLOB(0x7c9d98)<<< 250-DSN
Net::SMTP=GLOB(0x7c9d98)<<< 250-ETRN
Net::SMTP=GLOB(0x7c9d98)<<< 250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
Net::SMTP=GLOB(0x7c9d98)<<< 250-DELIVERBY
Net::SMTP=GLOB(0x7c9d98)<<< 250 HELP
Net::SMTP=GLOB(0x7c9d98)>>> MAIL FROM:<sender@nowhere.com>
Net::SMTP=GLOB(0x7c9d98)<<< 250 2.1.0 <sender@nowhere.com>... Sender ok
Net::SMTP=GLOB(0x7c9d98)>>> RCPT TO:<myself@myhost.com>
Net::SMTP=GLOB(0x7c9d98)<<< 250 2.1.5 <myself@myhost.com>... Recipient ok
Net::SMTP=GLOB(0x7c9d98)>>> DATA
Net::SMTP=GLOB(0x7c9d98)<<< 354 Enter mail, end with "." on a line by itself
Net::SMTP=GLOB(0x7c9d98)>>> MIME-Version: 1.0
Net::SMTP=GLOB(0x7c9d98)>>> From: sender@nowhere.com
Net::SMTP=GLOB(0x7c9d98)>>> To: myself@myhost.com
Net::SMTP=GLOB(0x7c9d98)>>> Subject: Testing email.
Net::SMTP=GLOB(0x7c9d98)>>>
Net::SMTP=GLOB(0x7c9d98)>>>
Net::SMTP=GLOB(0x7c9d98)>>> Just a test, please ignore
Net::SMTP=GLOB(0x7c9d98)>>> .
Net::SMTP=GLOB(0x7c9d98)<<< 250 2.0.0 r0EC3Qm3030991 Message accepted for delivery
Net::SMTP=GLOB(0x7c9d98)>>> QUIT
Net::SMTP=GLOB(0x7c9d98)<<< 221 2.0.0 myhost.localdomain closing connection

Perl script rectifying the encoding of a mixed UTF-8 / windows-1255 file

Suppose that some hodge-podge of scripts and template files create a single file, which has multiple encoding of Hebrew. That is, both UTF-8 and windos-1255. If the different encodings don’t appear in the same lines, the following script makes it all UTF-8:

#!/usr/bin/perl
use strict;
use warnings;

use Encode;
binmode(STDOUT, ":utf8");

while (defined (my $l = <>)) {
 eval { print decode("utf-8", $l, Encode::FB_CROAK); };
 next unless ($@);

 eval { print decode("windows-1255", $l, Encode::FB_CROAK); };

 if ($@) {
    print STDERR "Failed to decode line $.: $l\n";
    print $l;
  }
}

The binmode() call mutes warnings about wide characters (UTF-8) going to standard output.

The first decode() call does nothing if the handled line contains pure ASCII or UTF-8 encoded characters. The thing about it is the third argument, CHECK, which is set to FB_CROAK (man Encode for details). This tells decode() to die() if a malformed character is encountered. Being enclosed in an eval { }, it’s just a test. If this no-operation decoding goes by peacefully (that is, $@ is undefined) we know that the line was printed and one can go on to the next line.

If it does fail, a second attempt to decode() takes place, this time from the selected encoding. If it fails, the line is printed as is, and a warning is issued to standard output.

As a final note, it looks like this script could be improved by using FB_QUIET instead. This option makes decode() run as far as it can, and overwrites the input variable to contain the undecoded part. So this could be a method to munch through a string chunk by chunk, trying different encodings each time. Or so says the manual page. I never tried it.

 

Setting up a VPS server. It was a bumpy road.

Introduction

These are my own notes as I set up an OpenVZ VPS server, based upon CentOS 5.6 to function as a web and mailing list server. A $36/year 128 MB RAM machine was good enough for this.

Since there’s some criticism about the hosting provider, and it looks like they’re OK after all, I’m leaving their name out for now. The main purpose of this post is to help myself getting started again, if that is ever necessary (I sure hope it will never be).

Foul #1: Mails from hosting provider marked as spam

This is the first time it happens to me that automated emails from any service providers go to Gmail’s spam box. That includes the welcome mails as I subscribed, the payment confirmation and the message telling me the server is up. And messages about support tickets. None arrived.

Spamassassin gives these mails some points (1.3 or so) as well. I’ve hardly seen anything like this from any decent automatic mail producer. I first thought this was a major blunder, but then it turns out that machine-generated emails tend to get caught by spam filters. Since email messages that are relayed by a mailing list (mailman) don’t get caught, it looks like the spam filter checks the “received” chain of headers for the first hops of the message, and tries to figure out if that’s a decent ISP there. Just a wild guess.

Workaround: Add a filter in Gmail to never send emails from *@the-hosting-provider.com to the spam box. Simple, when you know about it.

Foul #2: 12 hours from payment to server running

Even for a low-cost service, 12 hours of “pending” is a bit too much. In particular when $36 have been paid. That alone filters out most scammers, I suppose.

Foul #3: Root password not set

Maybe I was naive to expect that the root password would be set in the server, so I tried to SSH the server with the password I had assigned during the subscription, but was consistently denied.

Workaround: Enter the VPS control panel and change the password.

Foul #4: Uncertified HTTPS link

The control panel of the VPS is accessed with a link to an IP address. Which is a bit weird, but let’s leave that alone. I mean, what about buying a domain for that purpose? To make things even worse, they supply an HTTPS link as well. Which works, but makes the browser display a scare “GET ME OUT OF HERE” message.

An uncertified HTTPS link is better than HTTP, even though cryptologists will argue that in the absence of a certificate, a man-in-the-middle attack is possible. But let’s get serious. It’s not really dangerous. It’s just yet another sign that they don’t give a shit. Setting up a domain and certifying it something you would expect from any serious company, just to avoid that scary warning message. But they didn’t.

Bump #1: Lacking yum repository

Among the first thing I did after logging in (because I’m addicted):

# yum install git
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror01.th.ifl.net
 * extras: mirror01.th.ifl.net
 * updates: mirror01.th.ifl.net
base                                                    | 1.1 kB     00:00    
base/primary                                            | 967 kB     00:00    
base                                                                 2725/2725

[ ... yada yada ... ]

vz-updates/primary                                      | 1.0 kB     00:00    
vz-updates                                                                 3/3
Setting up Install Process
No package git available.
Nothing to do

Are you kidding me? Using

# rpm -qa --last | head

I got a list of packages installed, many of which were marked with “el5″, which isn’t surprising,

# cat /etc/redhat-release
CentOS release 5.6 (Final)

since it’s a CentOS 5 distro (EL = Enterprise Linux).

The list of existing repos:

# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror01.th.ifl.net
 * extras: mirror01.th.ifl.net
 * updates: mirror01.th.ifl.net
repo id                                repo name                                      status
base                                   CentOS-5 - Base                                enabled: 2,725
extras                                 CentOS-5 - Extras                              enabled:   286
updates                                CentOS-5 - Updates                             enabled: 1,003
vz-base                                vz-base                                        enabled:     5
vz-updates                             vz-updates                                     enabled:     3
repolist: 4,02

So where’s git? On Fedora 12, I checked where git was loaded from (for comparison) with

$ yumdb info 'git*'
Loaded plugins: presto, refresh-packagekit
git-1.7.2.3-1.fc12.x86_64
 changed_by = 1010
 checksum_data = 470af233244731e51076c6aac5007e1eebd2f73f23cd685db7cd8bd6fb2b3dd1
 checksum_type = sha256
 command_line = install git-email
 from_repo = updates
 from_repo_revision = 1291265770
 from_repo_timestamp = 1291266900
 reason = user
 releasever = 12

[ ... here comes info about git-deamon and other packages ]

So CentOS’ repository doesn’t have git? That looks odd to me. A last try:

# yum list available | less

No, git wasn’t on the list. The fix was to add Repoforge to the list of repositories on the server (following the instructions):

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.i386.rpm
# rpm -i rpmforge-release-0.5.2-2.el5.rf.i386.rpm

And then “yum install git” went fine.

Bump #2: Bad host name

OK, it’s partly my fault. A short host name (without a dot) isn’t good enough. At least not for sending mails. A fully-qualified host name (such as example.com, as opposed to just “example”) is needed. Or sendmail starts up very slowly and then refuses to send mails.

Bump #3: Setting up reverse DNS

For the server to be able to send emails that aren’t immediately detected as spam, its IP must have the reverse DNS set to its host name.

There is a place to edit the rDNS name in the control panel (under “Network”) but it was disabled. Contact support, it said. So I did.

Having rDNS disabled by default is somewhat understandable to at least keep an eye on spammers. On the other hand, show me a spammer paying $36 upfront.

It took support 11 hours to answer this support request, asking me to supply the rDNS record I needed for manual setting. The actual fix came an hour later, so overall this was fairly OK.

The automatic feature is simply not supported. But it’s not like decent people need to change their rDNS every day.

Bump #4: No swap

It seems like there is no way to activate a swap file on the VPS server (in particular, losetup returns with “permission denied” so there is nothing to attach the swap partition to). So there’s no choice than to make sure that the overall memory consumption doesn’t exceed the allocation of virtual RAM, which is 128 MB in my case. Or processes will just die. I can understand the commercial sense in this limitation: If users would start putting large swap files on their systems, they would buy lower-cost machines and then complain that they’re not responsive.

The figure to keep track of is the amount of free + cached memory. For example,

$ cat /proc/meminfo
MemTotal:         131072 kB
MemFree:           47516 kB
Cached:            27156 kB
[ ... ]

The free memory is 47416 + 27156 = 74672 kB, which means 131072 – 74672 = 56400 kB is used. These are the figures that appear in the Control Panel.

Installation note: Setting up sendmail

By default, sendmail doesn’t accept external connections. Edit /etc/mail/sendmail.mc, changing

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

to

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl

This removes the restriction that only the local address is listened to. And also from

dnl DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl

to

DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl

(remove the “dnl” hence uncommenting the line).

Check with the hosting provider if they supply a mail relay server. Relaying through a well-reputed server can decrease the spam score of your mails. Besides, the hosting provider can decide to block all outgoing direct connections with mail servers out of the blue, because spams were flying out from their servers.

A line like the following should be added (possibly close to where SMART_HOST is mentioned in the file).

define(`SMART_HOST',`relay.myprovider.com')dnl

And then compile the file into sendmail.cf, and restart the server as follows:

# make -C /etc/mail
# service sendmail restart

Then test the server, possibly using a script. In particular, verify that the mail server isn’t relaying (accepting messages to other domains) or the server turns into a spam machine.

Installation note: Mailman

# yum install mailman

Then customize. See instructions here and also have a look at /usr/share/doc/mailman-2.1.9/INSTALL.REDHAT. There’s a need to access the host by its domain name (as opposed to just the IP address) so the local computer’s /etc/hosts may need to be fiddled with when working on a server not yet allocated with the new address.

Note: Do not edit /usr/lib/mailman/Mailman/mm_cfg.py for changing DEFAULT_URL_HOST and DEFAULT_EMAIL_HOST, if they happen to say

DEFAULT_URL_HOST   = fqdn
DEFAULT_EMAIL_HOST = fqdn

because in this case they’re set up automagically.

First, make sure that the mailman daemon is off:

# service mailman stop

To migrate a few lists from one server to another, copy the respective lists in /var/lib/mailman/{lists,archives} into the new server. Note that the “data” directory doesn’t contain any information on the lists, so therefore just adding these directories is enough.

The lists will not appear in the web interface if there was a domain switch during the list migration, as can be observed by searching for ‘web_page_url’ in the output of

# /usr/lib/mailman/bin/dumpdb config.pck

To fix this, go (for each list)

# /usr/lib/mailman/bin/withlist -l -r fix_url the-list-name

Make sure the files are owned by mailman with

# chown -R mailman:mailman ... the copied directories ...

At this point, the list should appear on the web console. Copy the entries into /etc/aliases, more or less like this:

## listname mailing list
listname:              "|/usr/lib/mailman/mail/mailman post listname"
listname-admin:        "|/usr/lib/mailman/mail/mailman admin listname"
listname-bounces:      "|/usr/lib/mailman/mail/mailman bounces listname"
listname-confirm:      "|/usr/lib/mailman/mail/mailman confirm listname"
listname-join:         "|/usr/lib/mailman/mail/mailman join listname"
listname-leave:        "|/usr/lib/mailman/mail/mailman leave listname"
listname-owner:        "|/usr/lib/mailman/mail/mailman owner listname"
listname-request:      "|/usr/lib/mailman/mail/mailman request listname"
listname-subscribe:    "|/usr/lib/mailman/mail/mailman subscribe listname"
listname-unsubscribe:  "|/usr/lib/mailman/mail/mailman unsubscribe listname"

And finally, turn the service on again:

# service mailman start

And make it a permanent service:

# chkconfig mailman on

Changing the subscription confirmation message (to tell users to look in the spam folder): Edit /usr/lib/mailman/templates/en/subscribe.html, and remove /var/lib/mailman/lists/{listname}/en/subscribe.html (if it exists, and contains nothing special for the list).

Then restart qrunner to flush cached templates:

# /usr/lib/mailman/bin/mailmanctl restart

See these two pages for more info about this (even though they’re not very accurate).

Fax to PC on HP Officejet 4500

I bought the Officejet 4500 to be able to send and receive faxes every now and then, use it as a scanner, and actually never print with it. I’m not sure I would recommend this to anyone. Not that it matters much, as it’s pretty phased out. Anyhow, the overall feeling about this machine and its software is that it looks for any excuse in the world to waste some ink. After all, ink is what HP makes its profits from.

The device (including its fax) refuses to do anything without ink cartridges with a good level of ink installed. Since I don’t intend to print at all, my main concern is that the ink will dry out, and I won’t be able to fax with it a few years from now, with no spare parts available.

Sometimes the machine won’t power up completely (with the exclamation sign blinking) unless there is paper loaded. It’s like the printer prepares to hijack some paper for something I never wanted it to do.

Many years ago, HP represented decent engineering. They’ve gone a long way since.

OK, to the point

First, FAX to PC must be activated on the PC side. In the HP solution center, click Settings, hover over “More Fax Settings” and pick “Digital Fax Setup Wizard”. This setting has most likely been completed during installation (a destination folder set up etc.).

If the Fax to PC feature has been (accidentally) disactivated on the printer, pick “Fax Settings” in the HP solution center, on the “Digital Fax Settings” tab make sure to activate the feature.

Then, on the printer, press the button with the wrench (“Tools”) > Fax Settings > Fax to PC and press the right arrow button until it says “Fax Print: Off” and then press OK. Don’t get confused by the part that says “Fax to PC: Off”: It isn’t really off, but it will be when pressing OK (and the fix above applies). An asterisk next to the setting indicates that it’s active, otherwise that state will become active only when pressing OK.

To receive a fax manually, press the left arrow as necessary to move the triangle to its leftmost position, press the green (“Start”?) button and the “2″ button for receiving a fax (with the phone off hook at least).

Remember that the HP Digital Imaging Monitor must be active for the fax to be received to the computer.

After a fax has arrived, a new file is silently generated in the designated destination folder (e.g. C:\fax). It’s a TIFF file with the name made up from the data and time the fax arrived (e.g. _20121217_114702.tif). No message appears on the screen.

If the computer is off, or the Monitor is off, the received fax will be pending in the machine without any message on the LCD display or anything indicating that action should be taken. The file is created as soon as the Monitor is activated (plus some 20 seconds or so). I don’t know how long the fax stays there, but I suppose turning off the fax machine will delete it.

It seems like there’s no support for Fax to PC for Linux. It looks like someone got fired at HP for supporting an ink-saving feature altogether.

I really miss my good old plain fax modem.

Signed arithmetics in Verilog: The only rule one needs to know

The golden rule is: All operands must be signed.

Verilog, it seems, is strongly inclined towards unsigned numbers. Any of the following yield an unsigned value:

• Any operation on two operands, unless both operands are signed.
• Based numbers (e.g. 12′d10), unless the explicit “s” modifier is used)
• Bit-select results
• Part-select results
• Concatenations

So the bottom line is to either use the $signed system function, or define signed wires and registers.

For example, to multiply a signed and unsigned register, yielding a signed value (of course), go something like this:

reg         [15:0] a; // Unsigned
reg signed  [15:0] b;
wire signed [16:0] signed_a;
wire signed [31:0] a_mult_b;

assign signed_a = a; // Convert to signed
assign a_mult_b = signed_a * b

Note that signed_a is one bit wider than “a”, so there’s room for the sign bit, which is always zero. If this wasn’t for this extra bit, a’s MSB would be treated as the sign bit in signed_a.

It may seem necessary to explicitly determine signed_a’s MSB with sometime like {1′b0, a} instead of just “a”, but the Verilog standard is pretty explicit about the signed vs. unsigned being determined by the expression only, and not by the left hand side. So “a” is treated as an unsigned value, and is hence extended by zero.

 

Permission denied to directory, despite group permission set OK

I tried to change directory to eli from other users belonging to the group “eli” and it failed with

$ cd ../eli/
-bash: cd: ../eli/: Permission denied

despite everything begin OK with the classic UNIX settings.

Reminder: After settings groups, there’s a need to either logout and login again, or use “su -” to refresh group settings. The “id” command reveals the effective group memberships.

It turns out that there’s another layer of settings, ACL (Access Control List), which is yet another way to make sure the computer is so protected that it drives you mad.

So let’s list the files:

$ ls -l
total 44
drwxrwx---+ 86 eli         eli          4096 2012-10-16 16:14 eli/
drwx------.  2 root        root        16384 2010-01-15 23:59 lost+found/

Note the ‘+’ and ‘.’ at the end of the “regular” permissions. What they tell us, is that there’s an ACL record on the “eli” directory. So effectively, the classic permissions are overridden. And this has nothing to do with SELinux, which is disabled on my computer.

Let’s see what we’ve got there:

$ getfacl eli
# file: eli
# owner: eli
# group: eli
user::rwx
user:qemu:--x
group::---
mask::rwx
other::--

So it means what it says: Despite the classic permissions, noone expect myself and qemu has permissions to the directory.

The remedy is to remove all ACL entries, and then set the permissions with chmod.

$ setfacl -b eli
$ ls -l
total 40
drwx------. 86 eli         eli          4096 2012-10-16 16:14 eli/
drwx------.  2 root        root        16384 2010-01-15 23:59 lost+found/
$ chmod g+xrw eli/
$ getfacl eli
# file: eli
# owner: eli
# group: eli
user::rwx
group::rwx
other::---

And now the system behaves like good old UNIX.

 

Workaround: “git push” from msysgit (git for Windows) hangs

I had set up my plain git-daemon and everything seemed to work fine, until I tried it from Windows. It just didn’t return from the command. According to a discussion in a newsgroup, the problem is a bug in msysgit’s implementation of side-band-64k, whatever that is.

Personally, even if an upgrade to msysgit existed, or if it was fairly easy to fix this and recompile, I wouldn’t want to take that path: That would require quite a few fixes to make in my case.

So my choice was to fix it on the server. The bug isn’t there, but I can stop the server from announcing that it supports this feature, so the client won’t even try. I don’t know what the impact of disabling this feature is, but it seems like it is about allowing status data for the impatient user to be sent multiplexed with actual object data.

In a xinetd setting, the main daemon calls runs git-daemon when a connection is made, which in turn calls “/usr/bin/git receive-pack” on a push request from the client. Just to make it clear, there is a git-receive-pack executable, but it’s not the one executed. Makes me wonder why it’s there at all.

The git program discloses its capabilities by sending a string saying something like “report-status delete-refs side-band-64k ofs-delta” on the TCP stream (on git version 1.7.2.3). So all that’s needed is to make sure the “side-band-64k” part is not transmitted. I suppose that will influence the way I’ll fetch from remote repositories in the future, but that’s a minor impact (I hope).

The sane way to fix this would be getting the source for git and recompile. I went for hacking the binary directly.

Namely, use XEmacs to open /usr/bin/git for hex editing (after making a backup copy, of course), find the place where that long capability string appears, and change “side-band-64k” to “side-bond-64k”. It’s exactly one byte to fix in the binary file.

And that does the trick. Not the most beautiful workaround, but quick and effective!

I suppose a similar manipulation would work on the client’s executable. But as mentioned above. it’s not effective for me.