Interactive Video Streaming

» Research » (Applications) Interactive Video Streaming


The latest development in wireless and hardware technology has enabled mobile devices to be used as multimedia consuming terminals. However, live streaming to mobile devices is still a challenging task due to their heterogeneity in hardware. This heterogeneity often requires media adaptation, or transcoding, to meet the requirements of each mobile device. In this project, we study a new architecture for mobile video streaming. First, we develop a novel video transcoding algorithm that leverages the scalability of Peer-to-Peer (P2P) systems. Next, we explore a low bit-rate video monitoring framework that detects illegitimate tampering that might occur in the P2P systems. We also conduct research on video streaming with interactive region-of-interest (IRoI). A high-resolution video is available at the source, but either bandwidth or display size prohibits streaming of the video at full resolution. This is particularly true for mobile devices. We therefore allow the user to interactively control pan, tilt, and zoom (PTZ) during playback and only deliver the portions of the bitstream needed to render the display at an appropriate resolution.

System Architecture

Mobile Streaming in P2P Systems

Video transcoding converts an original video bitstream to a new bitstream for a different encoding standard, smaller spatial resolution, reduced frame rate, or reduced quality (due to coarser quantization). However, transcoding poses a considerable computational burden on the streaming server because mobile devices often require individually customized transcoding. We achieve scalable and low-cost streaming for mobile users by leveraging the peer-to-peer architecture. In the proposed architecture, peers other than mobile users, called fixed nodes, contribute their computing power for distributed transcoding in a collaborative manner. Our scheme not only distributes the transcoding load among multiple nodes in the fixed part of the network, but at the same time provides a level of robustness that makes the scheme suitable for P2P live streaming. For example, even when some of fixed parent peers disappear, the video play-out at the mobile device still continues, albeit at a lower frame rate, thereby achieving graceful video degradation. The architecture is also ideally suited for wireless multi-homing and multipath routing by OpenFlow.

Video Quality Monitoring

Since intermediate peers transcode the original video to accommodate the heterogeneous capabilities of individual mobile users, many different versions of videos may exist in the P2P network. Some intermediaries might tamper with the video for many reasons, such as inserting unauthorized advertisements or piggybacking unauthentic contents. Therefore, quality estimation and tampering detection are important tasks and we address how to efficiently exchange the information to achieve these goals. We have developed a scheme to track the picture quality of video delivered to mobile devices by sending a small data volume (essentially a robust hash) to a remote monitoring server. Our new scheme is based on distributed source coding principles and outperforms the current ITU-T Rec. J.240 by more than an order of magnitude.

Interactive Region-of-Interest (IRoI) Streaming

One of our latest research topics is on video streaming with interactive region-of-interest (IRoI). A high-resolution video is available at the source, but either bandwidth or display size prohibits the streaming of the high-resolution video to handheld mobile devices. We propose a system that allows users to interactively control pan, tilt, and zoom (PTZ) during playback and deliver a smaller bitrate of video to them. This is familiar for browsing of high-resolution still (say satellite imagery and maps), and now supported for the first time for video in our ClassX streaming system. Our solution breaks the video into tiles at different resolution levels and simultaneously streams multiple sub-videos that make up the overall display.

Our system has recorded over 140 sessions in 8 courses and colloquia, and has served over 2000 online users. All sessions are recorded with static HD cameras and without human intervention. The system uses computer vision techniques to automatically track the lecturer and mimics the PTZ that a human camera operator would select. Usage statistics show that users select their own PTZ 75% of the time. Moreover, there is substantial diversity in the selected area, reflecting the individual users' interests. We also automatically recognize when slides are shown, using the techniques developed for mobile visual search, and synchronize the display of a slide deck. We have not yet implemented a mobile client for the system but hope to demonstrate IRoI streaming on a handheld device with a multi-touch screen during the next year. We also hope to explore and demonstrate the benefits of OpenFlow for this application, utilizing the new video quality monitoring technique. For more information, please visit ClassX.


Mobile Peer-to-Peer Streaming

The video on the mobile display (right) degrades gracefully as peers that support the mobile disappear (left).

Interactive Region-of-Interest (IRoI) Streaming

A user is demonstrating the virtual pan/tilt/zoom for a high-resolution (4K by 2K pixels) video using an air mouse.



Faculty Students
Bernd Girod
Jeonghun Noh
Derek Pang