Frisnit Electric Industrial Co. Ltd.



What have I done?

iPlayer for Kindle

| 9 Comments

Over the last few years the BBC‘s video on demand service iPlayer (a kind of British Hulu) has moved beyond desktop PCs, to games consoles, mobile phones, tablets, set top boxes and internet connected TVs. But there’s still a segment of the population who aren’t being catered for; those whose primary computing device is an eReader.

To address this I’ve designed a system to automatically convert a terrestrial digital TV transmission, as captured by a DVB card or PVR, to a sequence of static images annotated with the subtitles from the stream that convey the story as a kind of comic strip format. This can be viewed as HTML or PDF, so expanding the number of devices the content can be viewed on.

In this example I’ve converted an episode of Britain’s favourite aspirational soap, ‘Eastenders‘. The technical details of the conversion process are given later on in this post, but for now the basic operation of the system is discussed.

Principle
A basic strategy is to only render the frames where subtitles appear (subtitle frames). This is very simple to implement given a video file and corresponding timestamped subtitle file, and for some sequences this is quite adequate. For example:

However introductions and reaction shots often have no dialogue but may still contain key plot points. These will be missed by using the subtitle strategy alone. In a programme like Eastenders where the characters spend much of their time staring in to the distance in despair, these silent scenes must be included. To cope with this, scene detection software is used to produce a list of the start and end frames of each scene, then a snapshot is taken from the midpoint of each scene (scene frames). For example, here’s a sequence where Phil sneaks a look at somebody else’s mail after they’ve left the room:

To produce a complete rendering of the entire programme, the subtitle frames and the scene frames are combined. Then any scene frames from scenes that also contain a subtitle frame are discarded. A HTML file can now be created that simply presents the remaining frames in order. This HTML file can then be used as is, converted to a PDF for download on to an e-reader, or sent to a printer. This last format is not only for inveterate Luddites, it also allows the content to be consumed in areas where electronic devices are not suitable, for instance on the beach, or in an aircraft during take off and landing. This format also allows distribution via the postal service. With a typical PDF episode weighing in at only 20MB and with offline viewing, this format has a clear advantage over existing mobile iPlayer services.

Here’s the complete programme presented in an embedded player:

Eastenders, BBC1, Tue 31 May 2011 19:30 BST

Press the ‘Play’ button to advance to the next frame, and the other transport buttons to navigate the programme.

Technical details
In order to convert the video, first the DVB stream is demultiplexed in to the audio, video and subtitle streams using ProjectX. The DVB subtitles are actual images of the rendered captions carried as a subpicture stream. These could be used directly, however I had access to a pre-processed subtitle file in XML format. This gave a file with the caption text together with the start time, end time, position and colour of the caption. This was created from the subpicture titles using OCR and so occasionally suffers from misspellings and incorrect caption colours, but the XML format is extremely easy to process. An example XML subtitles file is given in the resources section of this post and more information about DVB subtitling can be found in the ETSI document ETSI 300 743 ‘DVB Subtitling Systems’.

Once the subtitles are available, a list of scenes in the programme is obtained. I used lav2yuv to do this, which works well enough for this demonstration. To use this first convert the MPEG file to MJPEG using ffmpeg:

ffmpeg -i source_video -r 25 -an -vcodec mjpeg destination_video.avi

Then build the scene list using lav2yuv:

lav2yuv -S list.txt destination_video.avi

This produces a file which is simply a list of start and end frames for each scene. Finally a simple PHP script is used to generate the final HTML output which can be passed through wkhtmltopdf to produce a PDF file. The whole process could easily be added to iPlayer’s current automated transcoding chain.

Future improvements

  1. The DVB stream also includes an audio descriptiontrack for the visually impaired which provides commentary about the scenes. Currently this only exists as audio, however if it were available as a timecoded text file similar to the subtitles file it would greatly improve the contextualisation of scenes. As an example, here’s the audio description immediately prior to the scene above with Masood:Example audio description
  2. It would be simpler and more accurate to use the DVB subpicture subtitles directly. Perhaps this can be done with the mplayer -sid option when capturing frames? However mplayer had difficulty capturing specific frames, preferring instead to capture the nearest keyframe.
  3. The scene detection software currently doesn’t work infallibly and can miss some scenes. It also requires the programme be converted to MJPEG before processing which is slow. A more efficient, tunable solution would be desirable.


Here’s the HTML version on a Kindle, inside a lovely case courtesy of Velvetmutineer

Resources
eastenders.xml – subtitles file
scene_list.txt – scene file generated by lav2yuv
eastenders.pdf – final PDF generated by wkhtmltopdf

The embedded HTML player shown above can also be viewed directly on a mobile device at http://cdn.frisnit.com/dvb2pdf/:

TwitterFacebookGoogle+LinkedInPinterestRedditStumbleUponEmail

9 Comments

  1. Wow, that’s some seriously creative thinking going on there! One thing that might improve the embedded player experience is to pre-load the next images. The loading indicator breaks the flow when ‘watching’. Cool work!

  2. At the very least, make the loading indicator more subtle… less like a 2×4 upside of the head. A little bitty spinner would convey the same idea without the brutality.

  3. Both good points. I have to admit the ‘player’ was created pretty quickly. Preloading and a nice fade between frames would definitely be an improvement!

  4. This is genius! Kudos for the approach! Any plan to productise this?

  5. Outstanding work. The potential for archived tv (and subtitled movies) is tremendous. It has educational and entertainment possibilities.

  6. Technically this is pretty cute and cool!

    But seriously, you can’t love your Kindle so much to want to watch a tv show in captioned frames in black and white? ;)

    I watch whole movies and tv series on my iphone during my train commute, it’s fantastic even on a small screen.

    Not taking away from the technical clevers in your innovation ;)

    HB

  7. Thanks for all your encouraging comments. I’ll post the script for doing the end to end conversion shortly. This started as a ‘because you can’ kind of project, but you never know where it’ll take you.

  8. This is an awsome idea. I teach Welsh as a second language for adults. As the BBC produce the Welsh soap Pobol y Cwm it’s available on i-player. I could use something like this in class for dialogue work as the programme is subtitled in both Welsh and English.

  9. Dyna syniad gwych! Unfortunately my local transmitter doesn’t carry BBC Cymru and for some reason subtitles don’t seem to be available for Pobol y Cwm on iPlayer at the moment. If you can get a dump of the DVB stream we could try it out.

Leave a Reply

Required fields are marked *.