Live TV / DVB on Linux demystified

Introduction

This is a not-so-short tutorial which is intended to make the setup of a Live TV media center on Linux a bit easier, walking through the processing chain from the digital transmission signal to the picture on the screen. Quite naturally, things go from “general knowledge” to a bit more hands-on. The approach here is to understand what you’re doing, something one can avoid when things just work by themselves. If you’re reading this, odds are they didn’t.

I suggest starting with the command line tools, as they give a lot of low-level information as they run. Once a TV channel can be viewed with these, I further suggest using Tvheadend as the backend, as it’s reasonably complicated to work with, and leaves a lot of control to the frontend software.

I also suggest a frontend (Kodi) for everyday TV viewing, and a way to configure it.

All in all, getting this to work is often a rather tedious process. This isn’t all that bad if one learns a few things on the way.

The DVB frontend (“adapter”)

Often with a USB interface, the DVB frontend receives the digital signal and turns it into a stream of bytes. Inside, it typically consist of a tuner, a demodulator and a USB interface chip. Often there’s a demux as well, which is discussed further on.

• The tuner (the analog part) nails down a piece of the frequency spectrum with the digital signal in it, and makes it accessible to the demodulator. The signal may arrive from a simple UHF antenna, a satellite dish or from a cable network. In principle, there is no difference: It’s an analog signal that carries a bitstream of several Mb/s, and the tuner’s job is to bring the signal down to a known lower frequency, where the demodulator expects it to be. The tuner cares only about the signal’s center frequency, and possibly its bandwidth.
• The demodulator (the digital communication part) turns the analog signal from the tuner into a stream of digital bits. This is the most sophisticated part, which includes a significant portion of signal processing and also decoding of error-correcting codes (and, of course, correcting bit errors if such are found and are correctable). All this is hidden from the end user, so all the demodulator tells us is typically if it’s locked on the signal or not. There are quite a few things that need to get synchronized properly, but all we get is something like “works” or “doesn’t work”. The demodulator may also supply information about the signal strength, the S/N ratio and the PostBER, which is an estimation of the bit error rate obtained after fixing bit errors by virtue of the error correction code. This estimation is possible because all but a fraction of the bits are recovered correctly, so the demodulator knows what its input signal would look like without noise and distortions, and so it can also tell how much noise comes in. And the S/N ratio is calculated accordingly.
• The USB interface chip (the computer hardware part) is the less interesting part, but it’s what the computer sees. The driver is often named after it, even though its the other devices that are important. Its main functionalities are: Relaying the output of the demodulator to the computer via a bulk USB endpoint, and supplying means to control and read status from the demodulator and tuner, which is almost always done with an I2C/SMBus over USB kind of bridge. The I2C bus interface may also be used to download firmware to these two devices.

In a Linux system, the DVB adapter is represented with device files in /dev/dvb/adapter0/ (the index goes up if there are several of them). The notable file is /dev/dvb/adapter0/dvr0, from which the data stream is read, possibly with plain file I/O. In other words, when the adapter is set up, it’s possible to record a proper video clip with just “cat”, as shown in this post. /dev/dvb/adapter?/{frontend?,mux?} are used to control the device.

There may be other device files as well, such as net0 (for controlling network packet functionality) or ca0 for Conditional Access.

For more insight, this set of slides may come handy.

What’s in the trunk?

Assuming that the DVB frontend is tuned and locked on a digital signal, there will be data in MPEG-TS format flowing out from /dev/dvb/adapter0/dvr0. Almost always, there are several TV/radio channels on a single digital transmission signal: The data stream is used to pass several types of packets, which may contain MPEG video or audio data, or other related data.

The packets that contain MPEG video or audio data are marked with an identifier, PID. In fact, watching a TV program consists of

• Tuning the DVB adapter to a certain reception frequency, and lock its demodulator on the digital signal
• Filter out all packets belonging to a certain PID, and pass them on to an MPEG video decoder
• Same for another PID, and pass them on to an MPEG audio decoder
• If there are subtitles, there’s another PID to filter out, and pass on to a subtitle rendering mechanism

So all in all, it’s a lot of packets multiplexed into a single stream of data, and it’s the receiver’s job to fish out those of interest. In order to make it easier, packets containing information that organizes the PIDs into services, i.e. TV and radio channels, are also transmitted on the same stream (in dedicated packets).

The end of this post shows the output of a scan with the dvbv5-scan command-line utility. It lists the information obtained in a specific digital stream in an organized manner. One thing that may be surprising about this list, is that a single service (i.e. TV channel) may contain more than a single audio PID. Which isn’t all that odd, as some TV channels may have alternative sound tracks, e.g. in different languages.

There’s also the dvbsnoop utility, which shows and dissects the packets of an MPEG-TS stream. Only for those interested in the really gory details.

By the way, in Tvheadend’s terminology, the raw stream of data that arrives from the demodulator is called a mux. This is a highly confusing misnomer, which probably came from idea that the packets in the stream are multiplexed. To the reset of the world, a “mux” is the machine that takes data from several sources and turns them into a stream. Which brings us to:

Demultiplexing

Assuming that the PIDs of the video and audio streams of the desired TV channel are known, there are two possibilities to filter them out:

• Software demuxing: Tell the DVB adapter to send everything to the dvr0 device file, and fish out the matching packets with the media player (or some intermediate software). Actually, it simply means that the media player ignores all packets that don’t have the requested PIDs.
• Hardware demuxing: Using the e.g. /dev/dvb/adapter0/demux0 device file to command the adapter to pass through only packets with certain PIDs to the /dev/dvb/adapter0/dvr0 output.

Software demuxing is the preferred choice for the typical domestic use, as it allows viewing more than one TV program at a time (assuming that both programs are on the same digital stream). Hardware demuxing is useful for viewing (or recording) TV with command-line utilities (see these two posts for examples of command line sessions).

Watching TV for real

Command-line utilities (see this, this, and this) can indeed be used to hack together some very basic kit for watching TV, and they are priceless for understanding why things went wrong with the fancier tools (hint: Somehow things always get wrong when video is involved).

But for everyday use, it’s best to let a TV streaming backend talk with the hardware. It allows fancy media center software as well as command line utilities to easily access TV channels. As of March 2017, I found Tvheadend + Kodi to be the best combination. In Kodi, I went for PVR IPTV Simple Client rather than Tvheadend’s own front end, as I’ll explain below.

So let’s first understand this backend / frontend business, and I’ll take Tvheadend as an example.

Tvheadend (formerly HTS TVheadend) is a TCP/IP server, which takes control of the DVB adapter(s). And it listens to two TCP/IP ports:

• HTTP Port 9981: Plain browsers can connect to http://localhost:9981/ for administration and configuration + machine readable playlists and Electronic Program Guide on certain URLs (see below). But most important: Availability of all TV channels as MPEG-TS streams, in a protocol easily accessible by a lot of software, with a plain HTTP connection.
• HTSP Port 9982: Home Tv Streaming Protocol (invented for Tvheadend?), seems to be used only be a handful of Linux clients. It’s a one-stop shop for all TV related information, but my own experience was a bit lame. So I’ll leave this aside for now.

The installation of Tvheadend hence involves making it work with the DVB adapter (which is usually simple if everything works smoothly with the command line utilities) as well as setting up the server. It’s may not be all that easy (there are worse), but it’s worth the effort (maybe my own messy jots will help), because:

• It allows simultaneous view of several TV channels, if they’re on the same digital stream (i.e. muxed on the same frequency channel). Watch one channel waiting for the commercials to end on another…
• View TV from any computer on the (wireless?) LAN.
• Virtually any media playing software supports its output format, MPEG-TS. Including stuff running on Windows.
• There are a lot of other features (Electronic Program Guide via the web interface with a browser, recording), but I’m not sure if these belong to the backend. But they may be useful to some.

So let’s take a look on how the Tvheadend conveys the TV channels to its clients. For this, I’ll assume that Tvheadend is properly installed, has been set up (through the web interface) to tune on some TV channels, and that it allows access without user/password from localhost (it’s a convenient setting, and it’s safe at least for 127.0.0.1/32). And that all access is done from the localhost (even though it can be any computer with HTTP access and due permissions. If so, replace “localhost” in the examples below with the IP or domain name of the server).

But first…

A word on IPTV / HLS (not really important for DVB, actually)

We’ll make a small detour to IPTV or HLS (HTTP Live Streaming), because Tvheadend does something similar. IPTV is the commonly used name for TV channels broadcast over the internet, whether it’s live or video-on-demand like kind of broadcasts.

An IPTV/HLS stream is essentially an MPEG-TS stream, similar to the DVB stream on air or cable. In order to make its broadcast over the web easier, it’s cut into chunks, each a few seconds long. The cuts are made on packet boundaries, so each chunk is a legal MPEG-TS segment by itself. A plain concatenation of several subsequent chunks (with e.g. “cat”) makes a perfectly playable MPEG-TS clip. Or stream.

Now to the IPTV client: To start off, the IPTV client is given an initial playlist (as a file or a URL to download this playlist from). That playlist is an M3U file, with one or several URLs, usually a TV channel for each. When the client accesses one of these URLs to start showing a channel, it often receives another playlist, which redirects it to other URLs, which in turn might redirect it further, and so on. These playlists are often set up to allow the client to choose different paths, depending on desired bit rate, display resolution, encoding format etc.

Eventually, possibly after a few redirection hops, the client ends with receiving a playlist containing a list of chunks, so it has the information on where to fetch chunks of MPEG-TS segments from. It starts fetching these chunks, concatenates then, and plays the video stream.

The complicated part of the HLS protocol is the traveling around playlists until the list of chunks is found. Once there, it’s just a matter of downloading those chunks, concatenating, and treating them as a DVB stream.

Tvheadend’s IPTV-like interface

So M3U playlists is the name of the game. Tvheadend offers the TV and radio channels it exposes as an M3U playlist, available at http://localhost:9981/playlist . In my case (Israeli DVB-T), it starts like this:

#EXTM3U
#EXTINF:-1 tvg-id="38e914f04571f2a3f5c915872ba6e794",88FM

http://localhost:9981/stream/channelid/1880418616?ticket=B0E6E9AB06F41C13C0AEC87B7A88966BCBCCE8F4&profile=pass

#EXTINF:-1 tvg-id="219e62923848dac382ed7fcd35c4ed9e",Aleph

http://localhost:9981/stream/channelid/308452897?ticket=88F2FD731008F28454AB8FF7F75BF896FA1F9C7F&profile=pass

#EXTINF:-1 tvg-id="fd874e5286a13d161eb1fa011fb42731",Bet

http://localhost:9981/stream/channelid/1380878333?ticket=3BF6D86889B5DB3DFCAF2EF25D07894B653C3700&profile=pass

#EXTINF:-1 tvg-id="aae608ee725cad880781301f68592dbc",Ch 1

http://localhost:9981/stream/channelid/1846077098?ticket=116EEC70D4201D82BF2A0F1E9AB7387EC9E12D30&profile=pass

#EXTINF:-1 tvg-id="41d97066c9c97e348f83269a6b18e8e6",Ch 10

http://localhost:9981/stream/channelid/1718671681?ticket=757FAE1BA26561DAEDB659E707627366A5071B17&profile=pass

#EXTINF:-1 tvg-id="fc2a79daaca0afd9a39123d4d0305a1f",Ch 2

http://localhost:9981/stream/channelid/1517890300?ticket=BE2E496685F892A1036B3C982888D0778784BDB4&profile=pass

[ ... etc ... ]

After the #EXTM3U header, there are pair of lines for each channel: The first line contains information about the channel (in particular the display name) and the second is the URL for accessing the channel. Unlike HTS/IPTV, this isn’t a go-find-another-playlist, but it directs immediately to where the video stream can be obtained.

The “tvg-id” tag is not common in playlist files in general, and it pairs the channel with its appearance in the EPG (more about that later). If you don’t have it, you probably have an old version of Tvheadend, which doesn’t support the EPG trickery I’m going to show below.

As the URLs in the playlist are “for real”, a plain wget command can be used to record any of these channels. For example, recording from Channel 10:

$ wget -O mytvshow.ts 'http://localhost:9981/stream/channelid/1718671681?ticket=757FAE1BA26561DAEDB659E707627366A5071B17&profile=pass'
--2017-03-13 11:25:59--  http://localhost:9981/stream/channelid/1718671681?ticket=757FAE1BA26561DAEDB659E707627366A5071B17&profile=pass
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:9981... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [video/mp2t]
Saving to: ‘mytvshow.ts’

mytvshow.ts           [           <=>      ]   3.10M   334KB/s

This will run in principle forever until stopped with CTRL-C. The mytvshow.ts file can be played with VLC, mplayer, ffplay or any other reasonable media player.

These URLs to the channels don’t change once Tvheadend has been set up. It’s therefore possible to download the playlist once, edit away unwanted channels, reorder the list (noted the radio channels at the beginning of the playlist above?), possibly combine with “real” IPTV channels, and feed a media center player with the edited playlist file.

It’s also possible to give these URLs directly to VLC and other media players. Viewing multiple channels at once is as simple as opening several instances of VLC.

One word about what Tvheadend does behind the scenes. In response to the wget command above, the following went to /var/log/syslog:

Mar 13 11:25:59 tv tvheadend[6410]: mpegts: 538MHz in Idan Plus T - tuning on Realtek RTL2832 (DVB-T) : DVB-T #0
Mar 13 11:25:59 tv tvheadend[6410]: subscription: 0018: "HTTP" subscribing on channel "Ch 10", weight: 100, adapter: "Realtek RTL2832 (DVB-T) : DVB-T #0", network: "Idan Plus T", mux: "538MHz", provider: "Idan +", service: "Ch 10", profile="pass", hostname="127.0.0.1", client="Wget/1.17.1 (linux-gnu)"

Note that the HTTP connection resulted in a “subscription” to a certain channel within Tvheadend. This reflects the way Tvheadend mediates its resources, a single DVB adapter in this case, to fulfill requirements of subscribers requesting services.

Consequently, stopping the “recording” (pressing CTRL-C) resulted in

Mar 13 11:26:11 tv tvheadend[6410]: subscription: 0018: "HTTP" unsubscribing from "Ch 10", hostname="127.0.0.1", client="Wget/1.17.1 (linux-gnu)"

Needless to say, something similar happens when a media player opens a connection for streaming live TV.

EPG

A neat feature of DVB is that data for an Electronic Program Guide (EPG) is often embedded in the digital stream, so the name of the current program, along with a short description, is available when zapping to a new TV channel. As well as a TV guide to past and future programs, directly on the TV, shown neatly by the media center software.

There is probably no need to configure anything in Tvheadend to make this work. All those EPG grabbers available are tools for transferring information into Tvheadend, in its absence from the digital stream itself. In particular, if there’s satisfactory information in the “Electronic Program Guide” tab in Tvheadend’s web interface (http://localhost:9981/ with a browser), nothing needs to be fixed.

The common format for exchanging EPG information in Linux is XMLTV, which as its name implies, is in XML format. Tvheadend exports it at http://localhost:9981/xmltv or http://localhost:9981/xmltv/channels (accessing the former will cause an HTTP 302 redirection to the latter).

As of March 2017, this doesn’t work on Tvheadend versions available on the “stable” apt repositories. If attempting to access the URL for XMLTV from a browser results in “1 Unknown Code” appearing, an upgrade is required. Or no EPG will be available with the setup I suggest below.

An XMLTV file typically looks something like this:

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">
<tv generator-info-name="TVHeadend-4.1-2405~geb495a0~xenial" source-info-name="tvh-Tvheadend">
<channel id="67a72084ee9a5ddb2fcd89129887bf78">
<display-name>Ch 99</display-name>
</channel>
<channel id="fa723385817605edc2b138d96c259b67">
<display-name>Ch 33</display-name>
</channel>
[ ... ]
<programme start="20170313113000 +0200" stop="20170313123000 +0200" channel="67a72084ee9a5ddb2fcd89129887bf78">
  <title lang="heb">ועדה ש.ח כת&apos; עב&apos; כת&apos; ער&apos;</title>
  <desc lang="heb">ועדה ש.ח כת&apos; עב&apos; כת&apos; ער&apos;
ועדה מיוחדת לפניות הציבור:
פניות ציבור בנושא התנהלות חברת החשמל בגביית תשלומים ומדיניות ניתוקי חשמל לצרכנים
</desc>
</programme>
<programme start="20170313121000 +0200" stop="20170313124500 +0200" channel="41d97066c9c97e348f83269a6b18e8e6">
  <title lang="heb">ראש בראש כת&apos; עב&apos; </title>
  <sub-title lang="heb">פרק 345</sub-title>
  <desc lang="heb">פרק 345
העיתונאי חגי סגל מארח ומתווכח. כת&apos; עב&apos;
</desc>
</programme>
<programme start="20170313124500 +0200" stop="20170313132000 +0200" channel="41d97066c9c97e348f83269a6b18e8e6">
  <title lang="heb">מעונן חלקית כת&apos; עב&apos; </title>
  <sub-title lang="heb">פרק 286</sub-title>
  <desc lang="heb">פרק 286
תחזית פוליטית: מבט אל השבוע הפוליטי והפרלמנטרי. מגישה: הדס לוי סצמסקי כת&apos; עב&apos;
</desc>
[ ... ]
</tv>

Note that the long hex blob marked red above matches the tvg-id entry of Channel 10 in the playlist given above. This allows pairing between an MPEG-TS stream and its info in the XMLTV file, and hence displaying the current TV program info for its respective channel.

Using Kodi as the front end

Kodi is a convenient front end for viewing TV on a media center computer, living room style. I suggest using the PVR IPTV Simple Client with a local file playlist, in particular because of the simplicity of this solution. And that it works so well.

The setup is fairly straightforward. First, install the plugin:

$ sudo apt-get install kodi-pvr-iptvsimple

and then, after having Kodi up and running, enable and set up the IPTV Simple Client as follows:

• Change setting level to Advanced
• System > Settings > Enable TV
• Enable and Configure PVR IPTV Simple Client (System > Settings > Add-ons > My add-ons > PVR Clients > PVR IPTV Simple Client). Set the playlist to local file, and pick one edited (as suggested above).
• Moving on to the EPG Settings tab, set Location to Remote Path, and XMLTV URL to http://localhost:9981/xmltv. As mentioned above, this requires a version of Tvheadend that supports XMLTV export. Check it manually with a browser.

The EPG interface isn’t necessary to watch TV properly, but makes Kodi display what’s on each channel in a neat way. As far as I know, the only alternative way to have EPG working with Kodi and Tvheadend is the Tvheadend Kodi plugin, which gave me errors all the time with the 4.09 version of Tvheadend.

Kodi has a screensaver enabled by default, which causes the screen to appear darker after a few minutes. It’s possible to turn it off under System > Settings > Appearance > Screensaver.

Summary

Use Kodi if you want it to look like a set-top-box, or vlc, ffplay or mplayer for a more computerish experience. Tvheadend gives a simple and robust interface to all of these, leaving the gory details to be forgotten. Once you’ve been through the setup, of course.

If Tvheadend doesn’t play ball, go for the command-line utilities.

And if all of this takes forever to complete, remember: TV is a waste of time either way.