tvheadend jots

This is a lot of random jots as I set up Tvheadend. As it was a bit of a battle, things may not be consistent below. There are some details below specific to my own environment (e.g. LAN addresses).

Environment: Linux Mint 18.1 running a 4.4.0-53-generic kernel, receiving DVB-T in Haifa, Israel with a HD-901T2 DVB-T2 USB stick.

Installation

This is the backend server for accessing the DVB tuner. Any software that is used to view live TV expects some kind of backend like this.

Add TVheadend repository, following this guide: The following file added as /etc/apt/sources.list.d/tvheadend.list:

deb https://dl.bintray.com/tvheadend/deb xenial stable

and loaded the key:

# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
Executing: /tmp/tmp.3BLJZuWTSs/gpg.1.sh --keyserver
hkp://keyserver.ubuntu.com:80
--recv-keys
379CE192D401AB61
gpg: requesting key D401AB61 from hkp server keyserver.ubuntu.com
gpg: key D401AB61: public key "Bintray (by JFrog) <bintray@bintray.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

and as usual,

# apt-get update

And install

# apt-get install tvheadend xmltv-util

ignoring the lack of verification (living on the edge):

WARNING: The following packages cannot be authenticated!
  tvheadend
Install these packages without verification? [y/N] y

xmltv-util was added since it was suggested when requesting the installation of tvheadend.

As part of the installation of Tvheadend, I was prompted for a username a password for the dedicated tvheadend user. This is the user/password pair for accessing the web interface, and not the Linux user which is generated during the installation (named “hts” was created, with no password assigned, i.e. impossible to login with password).

And then configure the server by connecting to port 9981 with a plain web browser.

Following this guide:

  • Went for the DVB Inputs tab, where /dev/dvb/adapter0 was found. Uhhh, there are three entries, two for Panasonic MN88473 (with DVB-C and DVB-T each) and an entry for Realtek RTL2832 (DVB-T). Picked the Realtek DVB-T #0 entry, as the Panasonic adapters can’t supply BER and SNR (and I killed it), as indicated in the debug output when this is attempted. Ticked the “enabled” checkbox. “Initial Scan” and “Idle scan” should be ticked as well. Click Save.
  • Went to the Networks Tab, and created a new DVB-T network, named it Idan Plus. Check “Skip Initial scan” (it slows down startup). Set Pre-defined Muxes to Israel: il-All.
  • Going back to the DVB Inputs tab, associate the DVB-T entry with the Network just created (drop-down menu for Networks). Click Save.
  • Under “Muxes”, monitor the scan status on the 538 MHz mux (for Haifa) should go from PEND to IDLE and the result to “OK”. It takes a couple of minutes or so before anything happens.
  • Disable the failed mux (not sure if it matters)
  • Go to Services, and find a nice list of them.
  • Map channels: On the Channel/EPG tab, click on the “Map Services” button, and accept the defaults.
  • Add an access entry for kodi: Set the username to e.g. kodiclient, set a password. Allow all interfaces except admin, set the streaming profile to htsp, and check all functionalities (video recorder to All DVR-rw).
  • Add an access entry for no-password users: Set the network prefix to “10.1.1.0/24;10.10.0.0/16;127.0.0.1/32″ with no credential request: Set the username to *, and the password to * (that is, a single “*” char each). Allow all interfaces, set the streaming profile to htsp, and check all functionalities (video recorder to All DVR-rw).

This is what a failed scan looks like (on tvheadend’s debug output):

2017-03-10 11:03:37.000 [   INFO] mpegts: 514MHz in Idan Plus - tuning on Realtek RTL2832 (DVB-T) : DVB-T #0
2017-03-10 11:03:37.000 [   INFO] opentv-ausat: registering mux 514MHz in Idan Plus
2017-03-10 11:03:37.004 [   INFO] subscription: 0002: "scan" subscribing to mux "514MHz", weight: 5, adapter: "Realtek RTL2832 (DVB-T) : DVB-T #0", network: "Idan Plus", service: "Raw PID Subscription"
2017-03-10 11:03:40.055 [WARNING] linuxdvb: Unhandled signal scale: 0
2017-03-10 11:03:40.069 [WARNING] linuxdvb: Unable to provide UNC value.
2017-03-10 11:03:42.000 [   INFO] mpegts: 514MHz in Idan Plus - scan no data, failed
2017-03-10 11:03:42.000 [   INFO] subscription: 0002: "scan" unsubscribing
2017-03-10 11:03:42.002 [   INFO] mpegts: 538MHz in Idan Plus - tuning on Realtek RTL2832 (DVB-T) : DVB-T #0
2017-03-10 11:03:42.002 [   INFO] opentv-ausat: registering mux 538MHz in Idan Plus
2017-03-10 11:03:42.009 [   INFO] subscription: 0004: "scan" subscribing to mux "538MHz", weight: 5, adapter: "Realtek RTL2832 (DVB-T) : DVB-T #0", network: "Idan Plus", service: "Raw PID Subscription"
2017-03-10 11:03:47.000 [   INFO] mpegts: 538MHz in Idan Plus - scan no data, failed
2017-03-10 11:03:47.000 [   INFO] subscription: 0004: "scan" unsubscribing

THIS IS THE SETTING FOR THE OLD Sin Hon TDH 601 / HD809 USB dongle. Just skip.

  • Went for the DVB Inputs tab, where /dev/dvb/adapter0 was found. Picked the DVB-T #0 entry (there was a DVB-C entry as well, but this is for cable TV frequencies). Ticked the “enabled” checkbox. “Initial Scan” and “Idle scan” should be ticked as well. Click Save.
  • Went to the Networks Tab, and created a new DVB-T network, named it Idan Plus. Uncheck “Skip Initial scan”. Set Pre-defined Muxes to Israel: il-All.
  • Going back to the DVB Inputs tab, associate the DVB-T entry with the Network just created (drop-down menu for Networks). Click Save.
  • Under “Muxes”, monitor the scan status on the 538 MHz mux (for Haifa) should go from PEND to IDLE and the result to “OK”. It takes a couple of minutes or so before anything happens.
  • Disable the failed mux (not sure if it matters)
  • Go to Services, and find a nice list of them.
  • Map channels: On the Channel/EPG tab, click on the “Map Services” button, and accept the defaults.
  • Add an access entry for kodi: Set the username to e.g. kodiclient, set a password. Allow all interfaces except admin, set the streaming profile to htsp, and check all functionalities (video recorder to All DVR-rw).

# service tvheadend stop

or, alternatively:

# systemctl stop tvheadend.service

(and start for turning it on). After stopping, running the service from console, as root:

# /usr/bin/tvheadend -f -u hts -g video

This launches a deamon (-f means “fork as daemon”), after changing the user ID and group as given by the flags.

To run it in the foreground, with debug messages to console,

# /usr/bin/tvheadend --trace --debug -d -u hts -g video

Which outputs during a successful scan of Idan Plus in Haifa:

2017-02-06 14:24:58.000 [   INFO] mpegts: 538MHz in Idan Plus - tuning on Silicon Labs Si2168 : DVB-T #0
2017-02-06 14:24:58.001 [   INFO] opentv-ausat: registering mux 538MHz in Idan Plus
2017-02-06 14:24:58.007 [   INFO] subscription: 0007: "scan" subscribing to mux "538MHz", weight: 5, adapter: "Silicon Labs Si2168 : DVB-T #0", network: "Idan Plus", service: "Raw PID Subscription"
2017-02-06 14:24:59.028 [WARNING] eit: invalid checksum (len 489, errors 1)
2017-02-06 14:24:59.698 [WARNING] mpegts: mux 514MHz in Idan Plus changed from DVBT freq 514000000 bw 8MHz cons QAM/16 hier NONE code_rate 2/3:NONE guard 1/4 trans 8k plp_id -1
2017-02-06 14:24:59.698 [WARNING] mpegts: mux 514MHz in Idan Plus changed to   DVBT freq 514000000 bw 8MHz cons QAM/16 hier NONE code_rate 2/3:2/3 guard 1/4 trans 8k plp_id -1
2017-02-06 14:24:59.700 [WARNING] mpegts: mux 538MHz in Idan Plus changed from DVBT freq 538000000 bw 8MHz cons QAM/16 hier NONE code_rate 2/3:NONE guard 1/4 trans 8k plp_id -1
2017-02-06 14:24:59.700 [WARNING] mpegts: mux 538MHz in Idan Plus changed to   DVBT freq 538000000 bw 8MHz cons QAM/16 hier NONE code_rate 2/3:2/3 guard 1/4 trans 8k plp_id -1
2017-02-06 14:25:03.000 [   INFO] mpegts: 538MHz in Idan Plus - scan complete
2017-02-06 14:25:03.000 [   INFO] subscription: 0007: "scan" unsubscribing
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0027/39] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0026/38] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0025/37] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0023/35] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0022/34] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0021/33] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0020/32] (missing in PAT/SDT)
2017-02-06 14:25:03.004 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 001F/31] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0014/20] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0013/19] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0012/18] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0011/17] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 0010/16] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000F/15] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000E/14] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000D/13] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000C/12] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000B/11] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: disabling service Idan Plus/538MHz/{PMT:0} [sid 000A/10] (missing in PAT/SDT)
2017-02-06 14:25:03.005 [   INFO] mpegts: 514MHz in Idan Plus - tuning on Silicon Labs Si2168 : DVB-T #0
2017-02-06 14:25:03.005 [   INFO] opentv-ausat: registering mux 514MHz in Idan Plus
2017-02-06 14:25:03.012 [   INFO] subscription: 0009: "scan" subscribing to mux "514MHz", weight: 5, adapter: "Silicon Labs Si2168 : DVB-T #0", network: "Idan Plus", service: "Raw PID Subscription"
2017-02-06 14:25:08.000 [   INFO] mpegts: 514MHz in Idan Plus - scan no data, failed
2017-02-06 14:25:08.000 [   INFO] subscription: 0009: "scan" unsubscribing

An .m3u8 playlist can be obtained with http://localhost:9981/playlist (or was it http://localhost:9981/playlist/channels?). Anyhow, if Tvheadend doesn’t like the URL, it goes “1 Unknown code”).

The playlist can be used with e.g. VLC connect to to play channels, possible several different channels from the same DVB adapter, if they share the same broadcast frequency ( = same mux in Tvheadend terminology).

Source digging

Grabbing the repository:

$ git clone https://github.com/tvheadend/tvheadend

Using “git blame” on src/webui/webui.c, it’s evident that Jaroslav Kysela added the /xmltv feature on Oct. 28th 2015 (as he said himself), and there are functional changes running as far as commit ID a87689803093e303ac4d458d35ac92c1bfc41f96 of Nov 11th 2015, which does XML escaping.

This post shows some JSON API as well.

Upgrade tvheadend

Since Kodi’s HTSP Tvheadend client worked horribly, I went for IPTV simple client. But in order to get the EPG feature working, I need a Tvheadend version that exports an XMLTV file through http. Ironically, the upgrade improved things with the HTSP client significantly, but there was still a need to double-click a channel twice to zap.

Change /etc/apt/sources.list.d/tvheadend.list to

deb https://dl.bintray.com/tvheadend/deb xenial unstable

and go

# apt-get update
# apt-get install tvheadend

Ignore warning that the package can’t be authenticated, and go ahead installing (upgrading, in my case). The new revision was 4.1-2405~geb495a0~xenial, according to “About” in the Web UI, which is git commit eb495a0984245c37a4474c7d3984acdd34edc071 from Jan 2nd, 2017. So not the absolutely latest, but definitely covers the /xmltv feature.

Files that were updated: /usr/bin/tvheadend, files under /usr/share/doc/tvheadend and a lot under /usr/share/tvheadend, and the man page (plus minus, this is based upon files an overlay filesystem).

Alternatively, this is the manual installation of the same. Look for tvheadend_4.1-2405~geb495a0~xenial_amd64.deb (it’s on my hard disk as well, under /storage/tv-mediacenter-stuff). Remove old version and install new:

# apt-get remove tvheadend
# dpkg -i tvheadend_4.1-2405~geb495a0~xenial_amd64.deb

This revision works great.

Some useful access URLs

Assuming that the server is on localhost:

  • Obtain m3U playlist of all live TV channels: http://localhost:9981/playlist
  • Obtain XMLTV file for EPG: http://localhost:9981/xmltv (or better, http://localhost:9981/xmltv/channels)
  • Watch TV channels directly from a browser (rather crude look, but works): http://localhost:9981/tv.html
  • Recorder log: http://localhost:9981/simple.html

And as mentioned above, there as several URLs for JSON formatted info on this post. I suppose I’ll add more as I come across these…

EDID info from mini HDMI2AV module (HDMI to RCA CVBS)

HDMI to AV converter (Composite Video + Audio on RCA plugs)

This is the information obtained with xrandr from the HDMI to AV converter  (Composite Video + Audio on RCA plugs) shown above:

Screen 0: minimum 8 x 8, current 1280 x 720, maximum 32767 x 32767
DP1 disconnected (normal left inverted right x axis y axis)
DP2 disconnected (normal left inverted right x axis y axis)
DP3 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
HDMI2 disconnected (normal left inverted right x axis y axis)
HDMI3 connected primary 1280x720+0+0 (normal left inverted right x axis y axis) 708mm x 398mm
 1280x720      60.00*+  59.94 
 1920x1080     60.00    59.94 
 1600x1200     60.00 
 1680x1050     59.88 
 1400x1050     59.95 
 1600x900      60.00 
 1280x1024     60.02 
 1440x900      59.90 
 1280x960      60.00 
 1280x800      59.91 
 1024x768      60.00 
 800x600       60.32 
 720x480       60.00    59.94 
 640x480       60.00    59.94 
VIRTUAL1 disconnected (normal left inverted right x axis y axis)

And with “verbose” flag:

Screen 0: minimum 8 x 8, current 1280 x 720, maximum 32767 x 32767
DP1 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x43
 Timestamp:  919743
 Subpixel:   unknown
 Clones:   
 CRTCs:      1 2
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
DP2 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x44
 Timestamp:  919743
 Subpixel:   unknown
 Clones:   
 CRTCs:      1 2
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
DP3 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x45
 Timestamp:  919743
 Subpixel:   unknown
 Clones:   
 CRTCs:      0
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
HDMI1 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x46
 Timestamp:  919743
 Subpixel:   unknown
 Clones:   
 CRTCs:      1 2
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 aspect ratio: Automatic
 supported: Automatic, 4:3, 16:9
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
HDMI2 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x47
 Timestamp:  919743
 Subpixel:   unknown
 Clones:   
 CRTCs:      1 2
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 aspect ratio: Automatic
 supported: Automatic, 4:3, 16:9
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
HDMI3 connected primary 1280x720+0+0 (0x11d) normal (normal left inverted right x axis y axis) 708mm x 398mm
 Identifier: 0x48
 Timestamp:  919743
 Subpixel:   unknown
 Gamma:      1.0:1.0:1.0
 Brightness: 1.0
 Clones:   
 CRTC:       0
 CRTCs:      0
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:
 EDID:
 00ffffffffffff00328d000000000000
 2d170103e87341782aee91a3544c9926
 0f505421080081008140818090409500
 a9c0b300a940011d007251d01e206e28
 5500c48e2100001ef339801871382d40
 582c4500c48e2100001e000000fd001e
 551e6414000a202020202020000000fc
 0048444d493241560a202020002001d6
 02031841458403020110230907078301
 000065030c001000011d007251d01e20
 6e28550010090000001e011d007251d0
 1e206e28550010090000001e8c0ad08a
 20e02d10103e96001009000000180000
 00000000000000000000000000000000
 00000000000000000000000000000000
 0000000000000000000000000000001d
 aspect ratio: Automatic
 supported: Automatic, 4:3, 16:9
 Broadcast RGB: Automatic
 supported: Automatic, Full, Limited 16:235
 audio: auto
 supported: force-dvi, off, auto, on
 1280x720 (0x11d) 74.250MHz +HSync +VSync *current +preferred
 h: width  1280 start 1390 end 1430 total 1650 skew    0 clock  45.00KHz
 v: height  720 start  725 end  730 total  750           clock  60.00Hz
 1920x1080 (0x4b) 148.500MHz +HSync +VSync
 h: width  1920 start 2008 end 2052 total 2200 skew    0 clock  67.50KHz
 v: height 1080 start 1084 end 1089 total 1125           clock  60.00Hz
 1920x1080 (0x10b) 148.352MHz +HSync +VSync
 h: width  1920 start 2008 end 2052 total 2200 skew    0 clock  67.43KHz
 v: height 1080 start 1084 end 1089 total 1125           clock  59.94Hz
 1600x1200 (0x114) 162.000MHz +HSync +VSync
 h: width  1600 start 1664 end 1856 total 2160 skew    0 clock  75.00KHz
 v: height 1200 start 1201 end 1204 total 1250           clock  60.00Hz
 1680x1050 (0x115) 119.000MHz +HSync -VSync
 h: width  1680 start 1728 end 1760 total 1840 skew    0 clock  64.67KHz
 v: height 1050 start 1053 end 1059 total 1080           clock  59.88Hz
 1400x1050 (0x13b) 101.000MHz +HSync -VSync
 h: width  1400 start 1448 end 1480 total 1560 skew    0 clock  64.74KHz
 v: height 1050 start 1053 end 1057 total 1080           clock  59.95Hz
 1600x900 (0x13c) 108.000MHz +HSync +VSync
 h: width  1600 start 1624 end 1704 total 1800 skew    0 clock  60.00KHz
 v: height  900 start  901 end  904 total 1000           clock  60.00Hz
 1280x1024 (0x117) 108.000MHz +HSync +VSync
 h: width  1280 start 1328 end 1440 total 1688 skew    0 clock  63.98KHz
 v: height 1024 start 1025 end 1028 total 1066           clock  60.02Hz
 1440x900 (0x119) 88.750MHz +HSync -VSync
 h: width  1440 start 1488 end 1520 total 1600 skew    0 clock  55.47KHz
 v: height  900 start  903 end  909 total  926           clock  59.90Hz
 1280x960 (0x11a) 108.000MHz +HSync +VSync
 h: width  1280 start 1376 end 1488 total 1800 skew    0 clock  60.00KHz
 v: height  960 start  961 end  964 total 1000           clock  60.00Hz
 1280x800 (0x11b) 71.000MHz +HSync -VSync
 h: width  1280 start 1328 end 1360 total 1440 skew    0 clock  49.31KHz
 v: height  800 start  803 end  809 total  823           clock  59.91Hz
 1280x720 (0x11f) 74.176MHz +HSync +VSync
 h: width  1280 start 1390 end 1430 total 1650 skew    0 clock  44.96KHz
 v: height  720 start  725 end  730 total  750           clock  59.94Hz
 1024x768 (0x122) 65.000MHz -HSync -VSync
 h: width  1024 start 1048 end 1184 total 1344 skew    0 clock  48.36KHz
 v: height  768 start  771 end  777 total  806           clock  60.00Hz
 800x600 (0x126) 40.000MHz +HSync +VSync
 h: width   800 start  840 end  968 total 1056 skew    0 clock  37.88KHz
 v: height  600 start  601 end  605 total  628           clock  60.32Hz
 720x480 (0x129) 27.027MHz -HSync -VSync
 h: width   720 start  736 end  798 total  858 skew    0 clock  31.50KHz
 v: height  480 start  489 end  495 total  525           clock  60.00Hz
 720x480 (0x12a) 27.000MHz -HSync -VSync
 h: width   720 start  736 end  798 total  858 skew    0 clock  31.47KHz
 v: height  480 start  489 end  495 total  525           clock  59.94Hz
 640x480 (0x12e) 25.200MHz -HSync -VSync
 h: width   640 start  656 end  752 total  800 skew    0 clock  31.50KHz
 v: height  480 start  490 end  492 total  525           clock  60.00Hz
 640x480 (0x12f) 25.175MHz -HSync -VSync
 h: width   640 start  656 end  752 total  800 skew    0 clock  31.47KHz
 v: height  480 start  490 end  492 total  525           clock  59.94Hz
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
 Identifier: 0x49
 Timestamp:  919743
 Subpixel:   no subpixels
 Clones:   
 CRTCs:      3
 Transform:  1.000000 0.000000 0.000000
 0.000000 1.000000 0.000000
 0.000000 0.000000 1.000000
 filter:

This was done with the switch in “PAL” position, but I’m not sure it matters.

Tvheadend starting slowly with a HD-901T2 DVB-T2 USB stick: The udev solution

The problem

Having A HD-901T2 DVB-T2 stick on a Linux Mint 18.1 running a 4.4.0-53-generic kernel, Tvheadend started painfully slow. It took more than a minute before the web interface was available. And here’s why:

Mar 11 10:22:35 tv systemd[1]: Starting tvheadend.service...
Mar 11 10:22:35 tv tvheadend[6019]:  * Starting Tvheadend tvheadend
Mar 11 10:22:35 tv tvheadend[6019]:    ...done.
Mar 11 10:22:35 tv systemd[1]: Started tvheadend.service.
Mar 11 10:22:35 tv tvheadend[6030]: main: Log started
Mar 11 10:22:35 tv tvheadend[6030]: config: loaded
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-T - loaded 43 regions with 1106 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-S - loaded 1 regions with 112 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: DVB-C - loaded 17 regions with 56 networks
Mar 11 10:22:35 tv tvheadend[6030]: scanfile: ATSC - loaded 2 regions with 14 networks
Mar 11 10:22:40 tv tvheadend[6030]: linuxdvb: adapter added /dev/dvb/adapter0
Mar 11 10:22:40 tv kernel: [ 5333.042417] usb 1-4: DVB: adapter 0 frontend 0 frequency 0 out of range (174000000..862000000)
Mar 11 10:24:23 tv tvheadend[6030]: linuxdvb: unable to open /dev/dvb/adapter0/frontend1

So that’s it: Tvheadend fails with /dev/dvb/adapter0/frontend1 somehow, and that failure takes time (the driver’s access timeouts?) and blocks the setup of other functionalities. The thing is that frontend1 is in fact the front end for the Panasonic MN88473 DVB-T/T2/C demodulator, and is redundant, because Tvheadend’s interface with the DVB stick is through /dev/dvb/adapter0/frontend0 (Realtek RTL2832).

The elegant solution is to make the offending device file inaccessible using udev. Since udev doesn’t rename device files that are created by the kernel for a long time, playing with its permissions is the way out. If Tvheadend is denied access to /dev/dvb/adapter0/frontend1, it won’t mess with it.

I have old post on udev hacking, by the way.

You may skip the following section, and go directly to “The fix”, if the gory details seem boring.

Analyzing the udev situation

Plugging in the DVB stick into the computer with

# udevadm monitor --udev --property

running makes a lot of output, but this among others:

UDEV  [90.760633] add      /devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1 (dvb)
ACTION=add
DEVNAME=/dev/dvb/adapter0/frontend1
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
DVB_ADAPTER_NUM=0
DVB_DEVICE_NUM=1
DVB_DEVICE_TYPE=frontend
ID_FOR_SEAT=dvb-pci-0000_00_14_0-usb-0_4
ID_PATH=pci-0000:00:14.0-usb-0:4
ID_PATH_TAG=pci-0000_00_14_0-usb-0_4
MAJOR=212
MINOR=4
SEQNUM=2829
SUBSYSTEM=dvb
TAGS=:uaccess:seat:
USEC_INITIALIZED=90759736

And looking at the device itself:

$ udevadm info -a -n /dev/dvb/adapter0/frontend1

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-3/dvb/dvb0.frontend1':
    KERNEL=="dvb0.frontend1"
    SUBSYSTEM=="dvb"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-3':
    KERNELS=="1-3"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="00"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="500mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 2"
    ATTRS{bcdDevice}=="0100"
    ATTRS{bmAttributes}=="80"
    ATTRS{busnum}=="1"
    ATTRS{configuration}=="USB2.0-Bulk&Iso"
    ATTRS{devnum}=="3"
    ATTRS{devpath}=="3"
    ATTRS{idProduct}=="0131"
    ATTRS{idVendor}=="15f4"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="astrometadvbt2"
    ATTRS{maxchild}=="0"
    ATTRS{product}=="dvbt2"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="removable"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="18880"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
    KERNELS=="usb1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{authorized}=="1"
    ATTRS{authorized_default}=="1"
    ATTRS{avoid_reset_quirk}=="0"
    ATTRS{bConfigurationValue}=="1"
    ATTRS{bDeviceClass}=="09"
    ATTRS{bDeviceProtocol}=="01"
    ATTRS{bDeviceSubClass}=="00"
    ATTRS{bMaxPacketSize0}=="64"
    ATTRS{bMaxPower}=="0mA"
    ATTRS{bNumConfigurations}=="1"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{bcdDevice}=="0404"
    ATTRS{bmAttributes}=="e0"
    ATTRS{busnum}=="1"
    ATTRS{configuration}==""
    ATTRS{devnum}=="1"
    ATTRS{devpath}=="0"
    ATTRS{idProduct}=="0002"
    ATTRS{idVendor}=="1d6b"
    ATTRS{interface_authorized_default}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{manufacturer}=="Linux 4.4.0-53-generic xhci-hcd"
    ATTRS{maxchild}=="7"
    ATTRS{product}=="xHCI Host Controller"
    ATTRS{quirks}=="0x0"
    ATTRS{removable}=="unknown"
    ATTRS{serial}=="0000:00:14.0"
    ATTRS{speed}=="480"
    ATTRS{urbnum}=="52"
    ATTRS{version}==" 2.00"

  looking at parent device '/devices/pci0000:00/0000:00:14.0':
    KERNELS=="0000:00:14.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="xhci_hcd"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x0c0330"
    ATTRS{consistent_dma_mask_bits}=="64"
    ATTRS{d3cold_allowed}=="1"
    ATTRS{device}=="0x22b5"
    ATTRS{dma_mask_bits}=="64"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="115"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{local_cpus}=="f"
    ATTRS{msi_bus}=="1"
    ATTRS{numa_node}=="-1"
    ATTRS{subsystem_device}=="0x1000"
    ATTRS{subsystem_vendor}=="0x1458"
    ATTRS{vendor}=="0x8086"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""

Only the first two entries are interesting: The others climb up through the USB port, the PCIebus and finally the PCI root port.

The fix

So I set up this file as /etc/udev/rules.d/50-dvb-rename.rules

SUBSYSTEM=="dvb", ATTRS{idVendor}=="15f4", ATTRS{idProduct}=="0131", KERNEL=="dvb*.frontend1", MODE:="0000", GROUP:="root"

And refreshed the udev system:

# udevadm control --reload

Don’t get discouraged by /etc/udev/ being almost empty: The systemd rules have moved to /lib/udev/, and following systemd conventions, local fixes should be in /etc/udev/.

That didn’t work exactly as I expected, but good enough (after replugging the DVB stick):

$ ls -l /dev/dvb/adapter0/
total 0
crw-rw----+ 1 root video 212, 0 Mar 11 10:50 demux0
crw-rw----+ 1 root video 212, 1 Mar 11 10:50 dvr0
crw-rw----+ 1 root video 212, 3 Mar 11 10:50 frontend0
c---rw----+ 1 root root  212, 4 Mar 11 10:50 frontend1
crw-rw----+ 1 root video 212, 2 Mar 11 10:50 net0

WTF? Note that the MODE assignment is with a := which is supposed to prevent overruling. And still, it’s group accessible. Where did that come from? Anyhow, that’s why the group is set to root. That fixed the problem with Tvheadend, which now finishes initializing 12 seconds after systemd announces its launch.

More udev rambling

I went on a bit on this. Nothing really for TLDRers.

Tesring (doesn’t really require root privileges, but the actions won’t take place):

# udevadm test -a add $(udevadm info -q path -n /dev/dvb/adapter0/frontend1)

or stating the path explicitly:

# udevadm info -q path -n /dev/dvb/adapter0/frontend1
/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
# udevadm test -a add /devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1
[ ... ]
GROUP 0 /etc/udev/rules.d/50-dvb-rename.rules:4
MODE 0 /etc/udev/rules.d/50-dvb-rename.rules:4
IMPORT builtin 'path_id' /lib/udev/rules.d/71-seat.rules:49
RUN 'uaccess' /lib/udev/rules.d/73-seat-late.rules:15
handling device node '/dev/dvb/adapter0/frontend1', devnum=c212:4, mode=0, uid=0
, gid=0
set permissions /dev/dvb/adapter0/frontend1, 020000, uid=0, gid=0
preserve already existing symlink '/dev/char/212:4' to '../dvb/adapter0/frontend
1'
created db file '/run/udev/data/c212:4' for '/devices/pci0000:00/0000:00:14.0/usb1/1-4/dvb/dvb0.frontend1'
[ ... ]

And here comes the big surprise: After the test, the frontend1′s permission was actually 0000 as required! So who’s fiddling with the permission after udev?

$ ls -l /dev/dvb/adapter0/total 0
crw-rw----+ 1 root video 212, 0 Mar 11 10:50 demux0
crw-rw----+ 1 root video 212, 1 Mar 11 10:50 dvr0
crw-rw----+ 1 root video 212, 3 Mar 11 10:50 frontend0
c---------+ 1 root root  212, 4 Mar 11 10:53 frontend1
crw-rw----+ 1 root video 212, 2 Mar 11 10:50 net0

If anyone knows, please comment below. For my current purpose, I could work this around by changing the group.

And if I’m at it: When Tvheadend runs, all device files in /dev/dvb/adapter0/ appear to have been deleted. They return when the Tvheadend is stopped. Does anyone have an idea on how and why?

Unlike what one might have expected, there was no need to update the initramfs. The attributes of frontend1 were the same (as the replug above, not the test) after a reboot:

# mount -o remount,rw /boot/
# update-initramfs -u -v

which didn’t make any difference, as expected.

Linux / DVB / Command line: Watching multiple channels with single DVB adapter

Some (or all?) DVB adapters can be set to submit the entire MPEG-TS stream to the host, without filtering specific channels (actually, PIDs). This allows viewing more channel at a time. This is demonstrated below with ffplay, even though ffplay tends to get stuck when it goes to a live stream.

It’s also not possible to view multiple channels with ffplay like this, because the device file can be opened only by a single instance of ffplay (or there’s a device or resource busy error). So in short, this is really just a proof of concept. Tvheadend does this anyhow. But the point there is to do it low-level style.

What I had: A HD-901T2 DVB-T2 stick on a Linux Mint 18.1 running a 4.4.0-53-generic kernel.

First, a scan should be done with dvbv5-scan, which creates a dvb_channel.conf file (shown in this post)

Using the -P flag to disable channel filtering:

$ dvbv5-zap -P -c dvb_channel.conf -r 'Ch 10'
using demux '/dev/dvb/adapter0/demux0'
reading channels from file 'dvb_channel.conf'
service has pid type 06:  2642 2641 2640
tuning to 538000000 Hz
pass all PID's to TS
  dvb_set_pesfilter 8192
       (0x00)
Lock   (0x1f)
Lock   (0x1f)
DVR interface '/dev/dvb/adapter0/dvr0' can now be opened
Lock   (0x1f) Signal= 92.55% C/N= 33.62dB postBER= 0

Note that the specification of ‘Ch 10′ in the command was merely a lazy way to set the frequency and other technical parameters.

Without stopping dvbv5-zap, go

$ ffplay /dev/dvb/adapter0/dvr0 2>log

and quit as soon as an image window appears. It’s a lot of mumbo-jumbo, but there’s a stream list somewhere in the middle — the example I captured is given at the end of this post.

So, to watch Channel 2:

$ ffplay -vst p:2 /dev/dvb/adapter0/dvr0

The p:2 specifier matches the “Program 2″ in the stream list. ffplay automatically matches the audio channel (even though -ast wasn’t specified).

Note that it’s ffplay that does the demuxing. This works (even better, actually) if the data stream is copied into a file first (with cat, for example) and then the file is played.

It also possible to handpick the streams with the PIDs, as listed in the dvb_channel.conf file (or in the stream list, as hex numbers in the square brackets). For example, watching channel 99:

$ ffplay -vst i:2689 -ast i:2690 /dev/dvb/adapter0/dvr0

or, equivalently

$ ffplay -vst i:0xa81 -ast i:0xa82 /dev/dvb/adapter0/dvr0

ffplay stream list

As captured in Haifa, Israel on March 10th 2017:

  Program 1
    Metadata:
      service_name    : ?Ch 1
      service_provider: ?Idan +
    Stream #0:30[0x401](heb): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:12[0xa01]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:31[0xa02]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 2
    Metadata:
      service_name    : ?Ch 2
      service_provider: ?Idan +
    Stream #0:0[0xa21]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:1[0xa22]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:2[0xa23]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:3[0xa30](heb): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:4[0xa31](rus): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:5[0xa32](ara): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
  Program 3
    Metadata:
      service_name    : ?Ch 10
      service_provider: ?Idan +
    Stream #0:6[0xa41]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:29[0xa42]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:28[0xa43]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:32[0xa50](heb): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:33[0xa51](rus): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
    Stream #0:34[0xa52](ara): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
  Program 4
    Metadata:
      service_name    : ?Ch 33
      service_provider: ?Idan +
    Stream #0:22[0xa61]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:23[0xa62]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 5
    Metadata:
      service_name    : ?Ch 99
      service_provider: ?Idan +
    Stream #0:11[0xa81]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:26[0xa82]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:35[0xa90](heb): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
  Program 6
    Metadata:
      service_name    : ?Ch 23
      service_provider: ?Idan +
    Stream #0:8[0xe01]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 12:11 DAR 15:11], 25 fps, 50 tbr, 90k tbn, 50 tbc
    Stream #0:27[0xe02]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 21
    Metadata:
      service_name    : Aleph
      service_provider: Idan +
    Stream #0:19[0xb01]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 22
    Metadata:
      service_name    : Bet
      service_provider: Idan +
    Stream #0:18[0xb11]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 23
    Metadata:
      service_name    : Gimmel
      service_provider: Idan +
    Stream #0:7[0xb21]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 24
    Metadata:
      service_name    : Dalet
      service_provider: Idan +
    Stream #0:9[0xb31]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 25
    Metadata:
      service_name    : Moreshet
      service_provider: Idan +
    Stream #0:25[0xb41]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 26
    Metadata:
      service_name    : 88FM
      service_provider: Idan +
    Stream #0:13[0xb51]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 27
    Metadata:
      service_name    : Musica
      service_provider: Idan +
    Stream #0:16[0xb61]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 28
    Metadata:
      service_name    : Reka
      service_provider: Idan +
    Stream #0:20[0xb71]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 29
    Metadata:
      service_name    : Galatz
      service_provider: Idan +
    Stream #0:24[0xb81]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 30
    Metadata:
      service_name    : Galgalatz
      service_provider: Idan +
    Stream #0:14[0xb91]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 36
    Metadata:
      service_name    : Radios
      service_provider: Idan +
    Stream #0:15[0xbf1]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 41
    Metadata:
      service_name    : Kol Barama
      service_provider: Idan +
    Stream #0:21[0xc41]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 42
    Metadata:
      service_name    : Lev HaMdina
      service_provider: Idan +
    Stream #0:17[0xc51]: Audio: aac_latm (HE-AACv2) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 45
    Metadata:
      service_name    : CLASSICAL bu
      service_provider: Idan +
    Stream #0:10[0xc81]: Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
  Program 7
    Metadata:
      service_name    : Idan+test_2
      service_provider: Idan+

So this list leaves no doubt that all channels were passed on to ffplay.

Linux / DVB: Playing with a HD-901T2 DVB-T2 USB stick

This is the successful version of a previous post, using a DVB USB stick that works, for a change.

IMPORTANT: If you’re using a computer with USB 3.0, consider connecting the DVB stick to the computer through an external USB 2.0 hub. In particular if things break after an hour or so of playback (see “USB disaster” below). This shouldn’t make any difference in theory, but solved the problem for me. Actually, this is the second time I solve a USB 3.0 issue with a USB 2.0 hub. If anyone has an idea on why this hub makes a difference, please post below. All USB cables were short, by the way.

Plugging in the card:

$ dmesg
[ ... ]
[ 2466.203556] usb 1-2: dvb_usb_v2: found a 'Astrometa DVB-T2' in warm state
[ 2466.287494] usb 1-2: dvb_usb_v2: will pass the complete MPEG2 transport stream to the software demuxer
[ 2466.287522] DVB: registering new adapter (Astrometa DVB-T2)
[ 2466.305556] i2c i2c-9: Added multiplexed i2c bus 10
[ 2466.305567] rtl2832 9-0010: Realtek RTL2832 successfully attached
[ 2466.310149] mn88473: module is from the staging directory, the quality is unknown, you have been warned.
[ 2466.313533] mn88473 9-0018: Panasonic MN88473 successfully attached
[ 2466.313551] usb 1-2: DVB: registering adapter 0 frontend 0 (Realtek RTL2832 (DVB-T))...
[ 2466.313718] usb 1-2: DVB: registering adapter 0 frontend 1 (Panasonic MN88473)...
[ 2466.322520] r820t 10-003a: creating new instance
[ 2466.335747] r820t 10-003a: Rafael Micro r820t successfully identified
[ 2466.335780] r820t 10-003a: attaching existing instance
[ 2466.343872] r820t 10-003a: Rafael Micro r820t successfully identified
[ 2466.351883] media: Linux media interface: v0.10
[ 2466.365085] Linux video capture interface: v2.00
[ 2466.388370] rtl2832_sdr rtl2832_sdr.1.auto: Registered as swradio0
[ 2466.388382] rtl2832_sdr rtl2832_sdr.1.auto: Realtek RTL2832 SDR attached
[ 2466.388386] rtl2832_sdr rtl2832_sdr.1.auto: SDR API is still slightly experimental and functionality changes may follow
[ 2466.395405] Registered IR keymap rc-empty
[ 2466.395640] input: Astrometa DVB-T2 as /devices/pci0000:00/0000:00:14.0/usb1/1-2/rc/rc0/input11
[ 2466.395932] rc0: Astrometa DVB-T2 as /devices/pci0000:00/0000:00:14.0/usb1/1-2/rc/rc0
[ 2466.401023] IR NEC protocol handler initialized
[ 2466.402794] IR RC5(x/sz) protocol handler initialized
[ 2466.405693] IR RC6 protocol handler initialized
[ 2466.408419] IR JVC protocol handler initialized
[ 2466.414092] IR Sharp protocol handler initialized
[ 2466.414716] IR MCE Keyboard/mouse protocol handler initialized
[ 2466.418597] input: MCE IR Keyboard/Mouse (dvb_usb_rtl28xxu) as /devices/virtual/input/input12
[ 2466.418906] usb 1-2: dvb_usb_v2: schedule remote query interval to 200 msecs
[ 2466.418972] lirc_dev: IR Remote Control driver registered, major 244
[ 2466.422821] rc rc0: lirc_dev: driver ir-lirc-codec (dvb_usb_rtl28xxu) registered at minor = 0
[ 2466.422827] IR LIRC bridge handler initialized
[ 2466.424978] IR XMP protocol handler initialized
[ 2466.426269] IR Sony protocol handler initialized
[ 2466.426274] IR SANYO protocol handler initialized
[ 2466.429076] usb 1-2: dvb_usb_v2: 'Astrometa DVB-T2' successfully initialized and connected
[ 2466.429300] usbcore: registered new interface driver dvb_usb_rtl28xxu

Seems pretty good. Judging by the device ID (15f4:0131 per lsusb), it’s exactly the device mentioned in Antti’s blog.

$ dvb-fe-tool
Device Realtek RTL2832 (DVB-T) (/dev/dvb/adapter0/frontend0) capabilities:
     CAN_FEC_1_2
     CAN_FEC_2_3
     CAN_FEC_3_4
     CAN_FEC_5_6
     CAN_FEC_7_8
     CAN_FEC_AUTO
     CAN_GUARD_INTERVAL_AUTO
     CAN_HIERARCHY_AUTO
     CAN_INVERSION_AUTO
     CAN_MUTE_TS
     CAN_QAM_16
     CAN_QAM_64
     CAN_QAM_AUTO
     CAN_QPSK
     CAN_RECOVER
     CAN_TRANSMISSION_MODE_AUTO
DVB API Version 5.10, Current v5 delivery system: DVBT
Supported delivery system:
    [DVBT]

Scanning went fine too:

$ dvbv5-scan /usr/share/dvb/dvb-t/il-All
Cannot calc frequency shift. Either bandwidth/symbol-rate is unavailable (yet).
Scanning frequency #1 514000000
       (0x00)
Scanning frequency #2 538000000
Lock   (0x1f)
Service Ch 1, provider Idan +: digital television
Service Ch 2, provider Idan +: digital television
Service Ch 10, provider Idan +: digital television
Service Ch 33, provider Idan +: digital television
Service Ch 99, provider Idan +: digital television
Service Ch 23, provider Idan +: digital television
Service Idan+test_2, provider Idan+: digital television
WARNING  Channel Idan+test_2 (service ID 7) not found on PMT. Skipping it.
Service Aleph, provider Idan +: digital radio
Service Bet, provider Idan +: digital radio
Service Gimmel, provider Idan +: digital radio
Service Dalet, provider Idan +: digital radio
Service Moreshet, provider Idan +: digital radio
Service 88FM, provider Idan +: digital radio
Service Musica, provider Idan +: digital radio
Service Reka, provider Idan +: digital radio
Service Galatz, provider Idan +: digital radio
Service Galgalatz, provider Idan +: digital radio
Service Radios, provider Idan +: digital radio
Service Kol Barama, provider Idan +: digital radio
Service Lev HaMdina, provider Idan +: digital radio
Service CLASSICAL bu, provider Idan +: digital radio

And some info:

$ dvb-fe-tool -g
FREQUENCY = 538000000
MODULATION = QAM/16
BANDWIDTH_HZ = 8000000
INVERSION = OFF
CODE_RATE_HP = 2/3
CODE_RATE_LP = 1/2
GUARD_INTERVAL = 1/4
TRANSMISSION_MODE = 8K
HIERARCHY = NONE
DELIVERY_SYSTEM = DVBT
$ dvb-fe-tool -m
       (0x00) Signal= 92.55% C/N= 32.98dB postBER= 0
       (0x00) Signal= 92.55% C/N= 32.98dB postBER= 0
       (0x00) Signal= 92.55% C/N= 32.98dB postBER= 0
       (0x00) Signal= 92.55% C/N= 32.98dB postBER= 0
       (0x00) Signal= 92.55% C/N= 32.98dB postBER= 0

Hmmm… Why isn’t it locked? And the data doesn’t change as I move the antenna around.

On the other hand,

$ dvbv5-zap -c dvb_channel.conf -r 'Ch 10'
using demux '/dev/dvb/adapter0/demux0'
reading channels from file 'dvb_channel.conf'
service has pid type 06:  2642 2641 2640
tuning to 538000000 Hz
video pid 2625
  dvb_set_pesfilter 2625
audio pid 2626
  dvb_set_pesfilter 2626
       (0x00)
Lock   (0x1f)
Lock   (0x1f)
DVR interface '/dev/dvb/adapter0/dvr0' can now be opened
Lock   (0x1f) Signal= 92.55% C/N= 38.67dB postBER= 0

Much better. I have a lock, and both the signal level and C/N change as I move the antenna. Actually, disconnecting the antenna and short-circuiting the input jack yields

Lock   (0x1f) Signal= 92.94% C/N= 28.19dB postBER= 1.01x10^-3

so with a momentary non-zero BER as shown above, so it looks like it’s legit.

So I have zero BER? I should see TV perfectly, then?

$ mplayer -cache 800 /dev/dvb/adapter0/dvr0
MPlayer 1.2.1 (Debian), built with gcc-5.3.1 (C) 2000-2016 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing /dev/dvb/adapter0/dvr0.
Cache fill: 16.00% (131072 bytes)   

libavformat version 56.40.101 (external)
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
TS file format detected.
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]

Oh no. What’s really weird is that going

$ cat /dev/dvb/adapter0/dvr0 > this

and in parallel

$ ffplay this

Actually does play TV more or less live (see this post for more on ffplay in this context). But

$ mplayer this
MPlayer 1.2.1 (Debian), built with gcc-5.3.1 (C) 2000-2016 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing this.
libavformat version 56.40.101 (external)
TS file format detected.
VIDEO MPEG2(pid=2625) AUDIO MPA(pid=2626) NO SUBS (yet)!  PROGRAM N. 0
MPEG: FATAL: EOF while searching for sequence header.
Video: Cannot read properties.
Load subtitles in ./

No sequence header? The clip is a minute long! Googling around, I found this tip, saying maybe there’s no PAT/PMT in the stream, and suggested adding “-demuxer lavf” flags to mplayer. That turned out to be corect:

$ mplayer -demuxer lavf this

Plays the clip, but

$ mplayer -demuxer lavf -cache 800 /dev/dvb/adapter0/dvr0

still complains about the cache being empty, like before. Only difference: When I hit CTRL-C, it plays the channel for a split second and closes. So I gave up this direction for now.

In hindsight, I don’t think I every tried to copy with my previous dongle. Maybe it worked, after all, even though there was a crucial difference: I never managed to get Tvheadend to display a picture either with the previous dongle. And with this one Tvheadend works like a charm.

I will never know, as I messed it up in the end.

USB disaster

It can start with a few like these:

Mar 10 10:44:21 tv kernel: [ 6133.433107] rtl2832 9-0010: i2c reg read failed -32

But after some time (an hour or so of playback)  ends with a full scale spontaneous disconnection of the USB device (without me touching anything, needless to say):

Mar 11 13:48:01 tv kernel: [ 1793.806742] usb 1-4: USB disconnect, device number 7
Mar 11 13:48:02 tv tvheadend[3325]: linuxdvb: Realtek RTL2832 (DVB-T) : DVB-T #0 - FE_READ_STATUS error No such device
Mar 11 13:48:02 tv kernel: [ 1794.805814] usb 1-4: dvb_usb_v2: rc.query() failed=-110
Mar 11 13:48:02 tv kernel: [ 1794.805885] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:02 tv acpid: input device has been disconnected, fd 7
Mar 11 13:48:02 tv acpid: input device has been disconnected, fd 6
Mar 11 13:48:02 tv kernel: [ 1794.933688] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:02 tv kernel: [ 1794.933706] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:02 tv kernel: [ 1794.933713] r820t 10-003a: r820t_write: i2c wr failed=-19 reg=0c len=1: f0
Mar 11 13:48:02 tv kernel: [ 1794.933720] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:02 tv kernel: [ 1794.937964] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:03 tv kernel: [ 1795.729720] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:03 tv kernel: [ 1795.772054] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:03 tv kernel: [ 1795.772069] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:03 tv kernel: [ 1795.772077] r820t 10-003a: r820t_read: i2c rd failed=-19 reg=00 len=4: f0 e1 2b ff
Mar 11 13:48:03 tv kernel: [ 1795.774643] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:03 tv kernel: [ 1795.933712] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:04 tv kernel: [ 1796.772180] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:04 tv kernel: [ 1796.772190] r820t 10-003a: r820t_read: i2c rd failed=-19 reg=00 len=4: f0 e1 2b ff
Mar 11 13:48:04 tv kernel: [ 1796.774426] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:04 tv kernel: [ 1796.933718] rtl2832 9-0010: i2c reg read failed -19
Mar 11 13:48:05 tv kernel: [ 1797.772322] rtl2832 9-0010: i2c reg read failed -19

sometimes along with a kernel warning regarding the USB hub work thread:

Mar 11 11:25:37 tv kernel: [  480.125459] INFO: task kworker/2:1:91 blocked for more than 120 seconds.
Mar 11 11:25:37 tv kernel: [  480.125485]       Tainted: G         C OE   4.4.0-53-generic #74-Ubuntu
Mar 11 11:25:37 tv kernel: [  480.125495] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 11 11:25:37 tv kernel: [  480.125506] kworker/2:1     D ffff880270613788     0    91      2 0x00000000
Mar 11 11:25:37 tv kernel: [  480.125547] Workqueue: usb_hub_wq hub_event
Mar 11 11:25:37 tv kernel: [  480.125561]  ffff880270613788 ffff880071e34a98 ffff880270f28c80 ffff8802705f8c80
Mar 11 11:25:37 tv kernel: [  480.125581]  ffff880270614000 ffff880272614108 ffff880071e34a98 ffff880071e34000
Mar 11 11:25:37 tv kernel: [  480.125599]  ffff880071e34aa0 ffff8802706137a0 ffffffff81831f75 ffff8802751a0800
Mar 11 11:25:37 tv kernel: [  480.125619] Call Trace:
Mar 11 11:25:37 tv kernel: [  480.125645]  [<ffffffff81831f75>] schedule+0x35/0x80
Mar 11 11:25:37 tv kernel: [  480.125695]  [<ffffffffc08adaf1>] dvb_unregister_frontend+0xd1/0x130 [dvb_core]
Mar 11 11:25:37 tv kernel: [  480.125718]  [<ffffffff810c3ec0>] ? wake_atomic_t_function+0x60/0x60
Mar 11 11:25:37 tv kernel: [  480.125745]  [<ffffffffc096f41d>] dvb_usbv2_exit+0x13d/0x3c0 [dvb_usb_v2]
Mar 11 11:25:37 tv kernel: [  480.125769]  [<ffffffffc096f714>] dvb_usbv2_disconnect+0x74/0xf0 [dvb_usb_v2]

After this, “lsusb” gets stuck with no output. Something really bad happened on the USB interface.

This renders the DVB stick useless. Luckily, there’s a simple solution — an external USB 2.0 hub between the DVB stick and the computer. I’m trying to figure out why this made a difference, but it did.

Linux kernel: /proc/iomem-like output to kernel log during boot

In order to have something similar to /proc/iomem printed out to the console (and dmesg), this piece of code can be implanted somewhere in the kernel code (in my case it was arch/arm/mach-zynq/common.c). It may be required to add #include headers, depending on the file it’s added to.

The original code that produces /proc/iomem is at kernel/resource.c.

static struct resource *adhoc_next_resource(struct resource *p, bool sibling_only)
{
  /* Caller wants to traverse through siblings only */
  if (sibling_only)
    return p->sibling;

  if (p->child)
    return p->child;
  while (!p->sibling && p->parent)
    p = p->parent;
  return p->sibling;
}

static void print_iomem(void)
{
  struct resource *r = &iomem_resource;
  int width = r->end < 0x10000 ? 4 : 8;
  pr_info("iomem mapping:\n");
  while (r) {
    pr_info("%0*llx-%0*llx : %s\n",
             width, (unsigned long long) r->start,
	     width, (unsigned long long) r->end,
	     r->name ? r->name : "");
    r = adhoc_next_resource(r, false);
  }
}

And then call print_iomem(). The output differs only from /proc/iomem in that it lacks the indentations representing the nesting of memory regions.

This was useful for me to get a snapshot during an early stage of Linux’ boot.

Linux: How to read through all files (checking a backup)

After finishing a backup to a USB stick, I like to verify that all is in place by reading through all files. Using the “tar” utility with verbose file output seemed to be a good idea. But…

Don’t

$ tar -cv . > /dev/null

For whatever reason, the files aren’t really read. Only the names of the files are printed.

It’s a bit of a mystery to me how the optimization was possible, but the time it took to complete this was way too short (using a USB stick flash)

Do

$ tar -cv . | cat > /dev/null

This, on the other hand, forces reading through the files

Linux Device Tree: What those status = “okay” are about

There are a lot of “okay” assignments in the kernel’s device tree. For example, arch/arm/boot/dts/zynq-zed.dts starts with

/dts-v1/;
#include "zynq-7000.dtsi"

and later on there’s, among others,

&sdhci0 {
	status = "okay";
};

&uart1 {
	status = "okay";
};

&usb0 {
	status = "okay";
	dr_mode = "host";
	usb-phy = <&usb_phy0>;
};

Let’s look on the last one, relating to the usb0 label. In zynq-7000.dtsi, there’s

		usb0: usb@e0002000 {
			compatible = "xlnx,zynq-usb-2.20a", "chipidea,usb2";
			status = "disabled";
			clocks = <&clkc 28>;
			interrupt-parent = <&intc>;
			interrupts = <0 21 4>;
			reg = <0xe0002000 0x1000>;
			phy_type = "ulpi";
		};

So the “okay” assignment in zynq-zed.dts overrides the one in zynq-7000.dtsi, “disabled”.

As explained on this page, if the “status” property is present, it must be “ok” or “okay”, or the device entry node is ignored (or should be). This allows the .dtsi file to include all possible hardware entries, and keep them “disabled”, and let the .dts file including it hand-pick those required, only by changing the “status”.

And indeed, in drivers/of/base.c it says

/**
 *  of_device_is_available - check if a device is available for use
 *
 *  @device: Node to check for availability
 *
 *  Returns true if the status property is absent or set to "okay" or "ok",
 *  false otherwise
 */
bool of_device_is_available(const struct device_node *device)
{
	unsigned long flags;
	bool res;

	raw_spin_lock_irqsave(&devtree_lock, flags);
	res = __of_device_is_available(device);
	raw_spin_unlock_irqrestore(&devtree_lock, flags);
	return res;

}
EXPORT_SYMBOL(of_device_is_available);

and

/**
 *  __of_device_is_available - check if a device is available for use
 *
 *  @device: Node to check for availability, with locks already held
 *
 *  Returns true if the status property is absent or set to "okay" or "ok",
 *  false otherwise
 */
static bool __of_device_is_available(const struct device_node *device)
{
	const char *status;
	int statlen;

	if (!device)
		return false;

	status = __of_get_property(device, "status", &statlen);
	if (status == NULL)
		return true;

	if (statlen > 0) {
		if (!strcmp(status, "okay") || !strcmp(status, "ok"))
			return true;
	}

	return false;
}

This feature was added to the kernel back in 2008 in commit 834d97d452208279edf11c57eca150360d2dd1d6, but it seems it took some time before it was actually adopted.

Note that when there’s no “status” entry, it’s treated as “okay”. This retains backward compatibility with DTS files that don’t care much about this feature.

bash: Jots on exiting the script

Trapping

Call a function (byebye in this case) just before exiting a script:

trap byebye exit

Or a command:

trap 'echo "Farewell cruel world!"' exit

It’s possible to catch various signals with “trap” as well.

Quit on error

In short: Have a

set -e

at the beginning of the script. It’s a global on-off feature, so it can be cancelled temporarily with something like

set +e
[ ... something we don't care if it fails ... ]
set -e

From bash’ man page, under the “-e” flag:

Exit immediately if a pipeline (which may consist of a single simple command), a subshell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces (see SHELL GRAMMAR above) exits with a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test following the if or elif reserved words, part of any command executed in a && or ││ list except the command following the final && or ││, any command in a pipeline but the last, or if the command’s return value is being inverted with !. A trap on ERR, if set, is executed before the shell exits. This option applies to the shell environment and each subshell environment separately (see COMMAND EXECUTION ENVIRONMENT above), and may cause subshells to exit before executing all the commands in the subshell.

From own testing:

  • Does quit if a function returns a non-zero value (as it says above)
  • If “set -e” appears in a file containing functions, it’s applies to the script invoking the function file (-e is global)
  • If an error occurs during a trap handler with set -e active, the trap script stops at that point (i.e. trap scripts are not exempt from set -e).

overlayroot (and permission issues) on Linux Mint 18.1

Introduction

Wanting my media center computer to maintain consistent behavior and be tolerant to sudden power outages, I looked for a way to have the root filesystem read-only. This can cause problems with programs that expect to write to all kind of files allover, and media related software has this tendency to break when things divert from exactly as expected. So overlayroot is an elegant solution, as the system behaves completely as usual, only the changes are stored in RAM, and vanish on the next boot.

Only risk: Working hard on some changes, just to discover that the system was in overlay mode, and all went poof.

My system is a Linux Mint 18.1 (”Serena”) which is a derivative of Ubuntu 16.04 (”Xenial”). It runs a 4.4.0-53-generic kernel.

Starting off

Following this excellent post:

$ sudo apt-get install overlayroot

Note that the installation generates a new initramfs in /boot with the update-initramfs command-line utility, but comparing with the old one I couldn’t find anything related to overlayroot.

Overlayroot isn’t active by default. It’s possible to edit /etc/overlayroot.conf, but since I’m planning to switch back and forth with GRUB, I might as well enable it with the Linux kernel command rather than disabling it with the same.

In order to create a GRUB2 menu entry, copy /etc/grub.d/10_linux into /etc/grub.d/15_linux_overlay, and change:

os="$1"

into

os="RAM OVERLAY $1"

but even more important, add the following line somewhere at the beginning of the file:

GRUB_CMDLINE_LINUX_DEFAULT="overlayroot=\"tmpfs:swap=0,recurse=0\" $GRUB_CMDLINE_LINUX_DEFAULT"

and run update-grub. Let’s break it down:

  • overlayroot=tmpfs:  obviously means use tmpfs for overlaying the root
  • swap=0: Disable swap. This is the default, but why rely on that
  • recurse=0: Don’t overlay any of root’s subdirectories, if mounted separately. /boot/, in my case. No reason to overlay it, as I’ve set it to mount read-only anyhow

By the way, there are empty “custom” entries in /etc/grub.d/, but since I want them exactly like the original, only with a couple of changes, copying those long scripts is more accurate.

To make this the default boot, change the GRUB_DEFAULT variable in /etc/default/grub to point to the desired index, or move 15_linux_overlay to say, 09_linux_overlay. And run update-grub, or course.

But then it didn’t work

When booting Linux with overlayroot enabled, the Avahi daemon failed to start. And indeed, trying to kick it off manually (as root):

# avahi-daemon -s
avahi-daemon: error while loading shared libraries: libavahi-common.so.3: cannot stat shared object: Permission denied

Sounds weird. Let’s look at the strace of the same command:

execve("/usr/sbin/avahi-daemon", ["avahi-daemon", "-s"], [/* 30 vars */]) = 0
brk(NULL)                               = 0x20a8000

[ ... yada yada ... ]

open("/usr/lib/x86_64-linux-gnu/libavahi-common.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@/\0\0\0\0\0\0"..., 832) = 832
fstat(3, 0x7fff3f2caaa0)                = -1 EACCES (Permission denied)
close(3)                                = 0
writev(2, [{"avahi-daemon", 12}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libavahi-common.so.3", 20}, {": ", 2}, {"cannot stat shared object", 25}, {": ", 2}, {"Permission denied", 17}, {"\n", 1}], 10) = 119
exit_group(127)                         = ?
+++ exited with 127 +++

WHATWHATWHAT? It’s fine to read from the file, but fstat() is denied?

And there was a similar problem with ping:

$ ping 1.1.1.1
ping: error while loading shared libraries: libcap.so.2: cannot stat shared object: Permission denied

Hint: If I moved /bin/ping to /bin/ping2, and ran ping2 instead, it ran normally.

Let’s talk about permissions

I know four types of permissions systems:

Since I issued the command as root, POSIX permissions won’t stop me. SELinux isn’t installed at all on Linux Mint 18.1, so maybe ACL?

$ cd /usr/lib/x86_64-linux-gnu
$ ls -l libavahi-common.so.*
lrwxrwxrwx 1 root root    24 Feb  3 13:00 libavahi-common.so.3 -> libavahi-common.so.3.5.3
-rw-r--r-- 1 root root 47952 Nov 24  2015 libavahi-common.so.3.5.3

$ getfacl libavahi-common.so.3
# file: libavahi-common.so.3
# owner: root
# group: root
user::rw-
group::r--
other::r--

$ getfacl libavahi-common.so.3.5.3
# file: libavahi-common.so.3.5.3
# owner: root
# group: root
user::rw-
group::r--
other::r--

Nope, nothing fishy about these files.

And this brings me back to the major hint from above: Changing the name of the executable made a difference. What functionality is sensitive to the executable’s path? Apparmor.

This gave me a major déjà vu from SELinux: That thing that’s supposed to make your system secure, but also causes mysterious failures until you’ve had enough of it, and go

$ sudo update-rc.d apparmor remove

Reboot, and then all was fine again. I have a feeling that many others will use this same command.

What it looks like

With overlayroot up and running, I get

$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                         3.9G     0  3.9G   0% /dev
tmpfs                        784M  9.5M  774M   2% /run
/dev/mapper/vg_ssd2-lv_root  118G  6.5G  106G   6% /media/root-ro
tmpfs-root                   3.9G  3.0M  3.9G   1% /media/root-rw
overlayroot                  3.9G  3.0M  3.9G   1% /
tmpfs                        3.9G  772K  3.9G   1% /dev/shm
tmpfs                        5.0M  4.0K  5.0M   1% /run/lock
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                    239M   99M  123M  45% /boot
cgmfs                        100K     0  100K   0% /run/cgmanager/fs
tmpfs                        784M   16K  784M   1% /run/user/1000

and

$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3987932k,nr_inodes=996983,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=801940k,mode=755)
/dev/mapper/vg_ssd2-lv_root on /media/root-ro type ext4 (ro,relatime,data=ordered)
tmpfs-root on /media/root-rw type tmpfs (rw,relatime)
overlayroot on / type overlay (rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
[...etc...]

So clearly, my SSD was mounted read-only, and used as “lowerdir”. The overlay directory is at /media/root-rw/overlay/, and the working directory (where files are prepared before going to the overlay, in order to ensure atomicity) is /media/root-rw/overlay-workdir/.

Preventing confusion

Trying to avoid confusion on whether the system is volatile or not, I added this script to the “Startup Applications” on my Cinnamon desktop (Menu > Preferences > Startup Applications):

#!/bin/bash

# Set a dedicated background for volatile (overlay) sessions

if grep -q overlayroot /proc/cmdline ; then
  /usr/bin/gsettings set org.cinnamon.desktop.background picture-uri \
    'file:///usr/local/share/wallpaper.jpg'

I put the script in /usr/local/bin/overlay-wallpaper, and the wallpaper as /usr/local/share/wallpaper.jpg. Since this script runs only when the system is volatile, this setting is gone when the system reboots. More about setting up Gnome attributes in this post.

FWIW, adding this script as a startup application resulted in a file named “.config/autostart/Special wallpaper for overlay session.desktop” which reads:

[Desktop Entry]
Type=Application
Exec=/usr/local/bin/overlay-wallpaper
X-GNOME-Autostart-enabled=true
NoDisplay=false
Hidden=false
Name[en_US]=Overlay wallpaper
Comment[en_US]=Special wallpaper for overlay session
X-GNOME-Autostart-Delay=0

Notes

  • overlayroot is a great way to tell what files are changed while the system is running. Just
    $ ls -RC on /media/root-rw/overlay/

    tells the whole story.

  • The size of tmpfs (and hence the overlay) is half the physical RAM by default. On my machine, that’s almost 4 GB, which is probably more than ever necessary. But if tmpfs becomes full, the computer will deadlock, as stated in the kernel’s documentation. Meaning: A simple program that decides to write a huge file can freeze the system completely. If that’s a problem, use some disk partition as the lowerdir, and wipe it clean (make2fs) on each boot before mounting.
  • If you want to save something permanently, it’s possible to remount the underlying root directory in read-write mode,
    $ sudo mount -o remount,rw /media/root-ro

    Note that this doesn’t disable the overlay — changes are not saved. To have something written permanently, change or create files under /media/root-ro/.
    This may or may not have an immediate effect on the currently seen root directory, depending on whether it has an entry in the overlay directory. So it’s useful mostly for minor fixes or to save something specific.

  • Every change in the file causes a copy into the overlay directory, including changes in permissions etc. If the file is deleted, a “whiteout” file is created in the respective position in the overlay, marking that it’s absent. Something like this (after deleting eli.tar.gz from my Desktop):
    $ cd /media/root-rw/overlay/home/eli/Desktop/
    $ ls -l
    total 0
    c--------- 1 root root 0, 0 Feb 21 13:48 eli.tar.gz
  • Judging by the vast amount of patches under fs/overlayfs/ in the kernel tree since v4.4 (to say, v4.10), the whole thing seems a bit shaky. On the other hand, who cares, if it messes up just pull the plug and reboot. And frankly, it works great.