Scope and purpose
Immersive media are gaining in popularity today, and significant efforts are being undertaken in academia and industry to explore its immanent new scientific and technological challenges. There are significant activities in industry and standardization to provide enablers for production, coding, transmission, and consumption of this type of media and the new user experiences it enables. In terms of standardization, the topic has triggered multiple activities in the areas of systems, 3D graphics, audio, image, and video. The technological roadmap foresees an evolution from consumption of the visual media with three degrees of freedom (so called “3DoF”, ability to look around at a fixed viewing position in an observed scene, i.e. 360° video) to 3DoF+ (enabling limited modifications of the viewing position), to different variants of 6DoF (six degrees of freedom, allowing the user not only to look around but also to move around in the observed scene). Different terminology is used in the various communities, referring to immersive or omnidirectional media, virtual reality (VR), or specifically, to 360° video. While the coded representation of audio-visual media for 6DoF is a field of very active research, 3DoF technologies have sufficient maturity to progress towards specification in near-term standards and recommendations. At the video codec level, this includes coding of 2D and 3D virtual reality (VR) / 360° content using the HEVC standard (Rec. ITU-T H.265 | ISO/IEC 23008-2) as the initial step, with new Supplemental Enhancement Information (SEI) messages for omnidirectional video. For audio, as an example, the already published ISO/IEC 23008-3 MPEG-H audio provides all enablers for 3D audio including channel, object and scene-based representations as well as 3D rendering. For storage and delivery, the Omnidirectional Media Format (OMAF) (expected to be published by early 2018 as ISO/IEC 23090-2) provides a set of consistent enablers for download and OTT streaming of 3DoF content. At the same time, immersive still image coding formats are currently developed in JPEG Pleno.
With respect to video, ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are currently preparing for standardization of video coding technology with a compression capability that significantly exceeds that of the HEVC standard and its current extensions. Such future standardization could take the form of additional extension(s) of HEVC or an entirely new standard. The scope of this joint activity includes consideration of a variety of video sources and video applications, including camera-view content, screen content, consumer generated content, high dynamic range content, and also explicitly virtual reality/360° content. A Joint Call for Proposals on Video Compression with Capability beyond HEVC is going to be published by ITU-T and MPEG in October 2017 with responses to be evaluated in April 2018. The standardization timeline foresees the finalization of the specification by the end of 2020.
On the systems side, and beyond the initial 3DoF activities, the new MPEG project on immersive media (referred to as MPEG-i) attempts to provide a more consistent view on immersive media and support new experiences in the mid and long-term. Based on use cases beyond 3DoF and architectural considerations, audio, video and 3D graphics aspects are evaluated, including new representation formats such as point clouds and light fields. Orthogonal aspects such as improved delivery, consistently reported metrics as well as quality evaluation of immersive media are also in scope.
Also outside of MPEG, the first set of enabling specifications for immersive media with 3DoF are cornerstones for evolving systems and related standardization and interoperability activities. Among others, 3GPP recognizes the value of MPEG 3DoF technologies for the work on 5G VR Streaming that is in progress and expected to be finalized by mid of 2018. The VR Industry Forum promotes the MPEG enablers for full end-to-end operability, combining them with production, security, distribution and rendering centric activities. On the latter, in particular the work in Khronos/OpenXR and W3C WebVR groups target to provide interoperability for platform APIs.
This Special Issue aims at a capture of the status of this emerging technology with respect to latest scientific progress, to corresponding standardization efforts, to subjective assessment of immersive media and also with respect to the impact of this technology on regular users. Original and unpublished research results with topics in any of the following areas or beyond are hereby solicited.