Live events are an integral part of our society. From classical concerts to festivals, people want to gather impressions with their own senses. Not only the current COVID-19 pandemic but also ecological and practical life factors, such as traveling for too long or the lack of childcare in the evening, mean that many people cannot physically attend events. This is precisely where technical solutions come in, making an audience experience possible through audiovisual immersion that comes much closer to “being there live” than previously standard formats.

The rapid development of digital media formats and offerings in combination with increasingly powerful playback devices such as smartphones or tablets offers a wide range of opportunities for digitally supplementing traditional performance practices, for example at concerts, plays, or other event formats such as congresses or trade fairs. In addition, digitization also offers extensive opportunities for developing new, independent formats and addressing new target groups.

The complexity of new technologies and the often associated high implementation costs for their online platforms make it difficult for cultural professionals to offer their services at the current state of the art. The three institutes Fraunhofer FOKUS, Fraunhofer HHI and Fraunhofer IIS have launched the project Virtual LiVe. In this context, Virtual LiVe aims to develop a platform that provides suitable and scalable high-end solutions for the event streaming needs of different industries in the form of a modular system. From this modular system, SMEs can put together a customized program for their individual event needs and thus obtain a technology-based solution to realizetheir formats. Existing offerings should thus be significantly surpassed in terms of quality, accessibility (payment barriers, hardware integration, the prerequisite of high Internet speed, etc.), and legal aspects (especially concerning the topics of data sovereignty and EU data protection directives).

Innovative media technologies provided by Fraunhofer FOKUS will play a central role in enabling low-latency adaptive bitrate streaming via content delivery networks and using streaming formats such as MPEG-DASH and CMAF. The standard-compliant MPEG DASH end-to-end solution FAMIUM DASH will be used for this purpose on a variety of platforms like Mobile, Web, TV and VR. The dash.js open source reference player maintained by Fraunhofer FOKUS will be used as an underlying player on all Web-based platforms. With low-latency mode activated, seamless interaction and audience feedback can be achieved during live events. In addition, the FAMIUM SAND metric reporting and streaming analytics tool will be integrated in the Virtual Live platform to monitor the streaming performance and QoE.

Additionally, Fraunhofer FOKUS researchers worked on the immersive visualization of the novel media formats in omnidirectional projections, using VR/AR glasses or on mobile devices. In this way, concerts can be broadcast as a 360° live stream in planetariums. Viewers not present on-site can interactively control and move 360° cameras on camera cranes. The spectators themselves can select the viewing angle.

Viewers equipped with VR glasses can “virtually” immerse themselves in the middle of the action during a live concert and thus, for example, take part in the show in the middle of the stage. Through a 3D 360° camera on stage, they receive the comprehensive view streamed into the VR glasses in real-time at home. They have the feeling of actually being on stage. Through a webcam or the sensors of the VR glasses and a VR controller, the movements and gestures from the spectator at home can be transmitted to the stage so that the spectator is visualized on a monitor on stage (via live video or 3D avatar). In this way, he can dance or play along (e.g. virtual drums, (air) guitar, etc.) and interact directly with the musicians (e.g. headbanging, high-fiving, etc.).

Virtual LiVe is hosted at the 3IT Innovation Center and funded under the Fraunhofer-Gesellschaft's “KMU-akut” program “Forschung für den Mittelstand (Research for SMEs)”.