Immersive Visualization

A multi-projector system is often necessary when projecting content on curved screens, such as panoramic or dome projections. The challenge lies in calibrating the projectors with reasonable expenditures of time and effort in such a way that a homogeneous image is created.

Fraunhofer FOKUS has developed an automatic camera-based projector calibration procedure that stands out with its great flexibility and speed. Test patterns are projected onto the screen and automatically captured by digital cameras.

Using imaging processing algorithms, the image parts sent by each projector are then distorted and adapted to the screen in such a way that a high-resolution, seamless image emerges. The projectors' partial images are accurately adapted to one another in the areas of overlap. Automatic projector re-calibration can easily and quickly be run whenever the position or the parameters of one or more projectors change. The projection screens can be of any shape: domes, spheres, cylinders, and curved screens are just some examples.

The system thus covers all necessary functions for an automatic multi-projector calibration: geometry correction (warping), automatic softedge blending, brightness correction and projector auto-alignment. Fraunhofer FOKUS holds several patents on camera-based projector autocalibration.

Fraunhofer FOKUS is licensing the software (warping, blending, and projector auto-calibration) and integrates it in existing and new systems, such as media servers, warping boxes and projectors.

In the last ten years, more than 130 multi-projector installations with more than 600 projectors in 13 different countries worldwide have been created using the Fraunhofer FOKUS software for projector autoalignment, warping and edgeblending. Most of the installations are planetarium domes, amusement parks, flight simulators and 3D dome cinemas.

In 2013, Fraunhofer FOKUS disclosed the details of their automatic projector calibration technology in the book “Fullspace-Projektion. Mit dem 360°Lab zum Holodeck” that was published by Springer Publishing.

The screen configurator is the planning tool. With it, the projection geometry is defined in advance and an appropriate projector setting considering the respective projector types and determining projector number, position, and orientation is determined. It is responsible for auto-calibration and calculates the distortion rules needed by the player to compose a seamless overall image regarding geometry, brightness, and color from the projectors’ partial images.

Different media types are combined to shows and replayed in real-time with a resolution of 8K x 8K. Similar to a video editing program, various media types such as film and still images can be combined and external devices can be integrated. Gamepads or joysticks can be connected to the system to provide an immersive gaming experience. Additionally, device-free control of multi-media content is possible through gesture commands. Live input can easily be integrated through the combination of different media in real-time (Realtime Compositing).

The technology is suitable for active and passive stereo projection.

The software can be licensed in two ways: the software can either control multi-projector systems directly as a stand-alone model. Alternatively, individual segments of the system can be integrated into existing visualization systems. Fraunhofer FOKUS is available to license holders as a competent partner in the further development of the system. Carl Zeiss is using the system as the basis for its PowerDome system (digital planetarium).

Depending on the application, the applications run on various smartphones, tablets, AR glasses (e.g. Microsoft HoloLens, Meta2) or VR glasses (e.g. Oculus Rift, HTC Vive).

By means of projections, AR can even be implemented without any additional equipment on the part of the user. The technology has already been used in various projects:

• ARMARE - Augmented Reality Telepresence System for the Deep Sea
• Visualization of complex graph structures using VR technologies

The scientists of Fraunhofer FOKUS are working on technologies for augmented, mixed and virtual reality applications. These are summarized under the term Extended Reality (XR). They advise their customers on current technologies for XR applications and on suitable devices for customer-specific use. In addition, they develop and implement prototypes as well as Apps and programs for the control of VR, AR and MR technologies. Their services cover the entire value-added process.

Crystal-clear images, top-quality surround sound, 3D films, viewers up close to the action – those are the prospects offered by the cinema of the future. Its forerunners are 3D and Imax cinemas, which continue to use analogue film technology but project high-quality images onto curved projection surfaces, giving viewers the impression of being in the middle of the action. But the thrills the technology offers come at a price. Such cinemas require special projectors weighing several tons. Complex and expensive technologies are needed for filming and projection, which is why the films are generally short and few in number and why ”dome cinemas” are still quite a rarity. The only place where full digital dome projection has so far really caught on is in planetariums, and in recent years a variety of technologies have become available for this purpose.

Fraunhofer FOKUS has developed a new solution for digital dome projection in close cooperation with Carl Zeiss. It allows digital images and videos to be displayed on a concave hemisphere using standard projectors and commercially available PCs – without loss of quality even in the case of high-resolution images. Image distortion correction, crossfading and the synchronization of the different components on the dome are done in real time.

Image Distortion Correction in Real Time

Today, showing digital films on curved projection surfaces involves a lot of preliminary work. Before projection commences, the individual scenes are put together on the computer using a video-editing program, and the entire video is computed. During this process, the partial images are distorted to adapt them to the curved projection surface and make them produce a realistic overall image on it. But a film remains correctly adjusted with respect to the screen only until there is a change in the position of the projector or the structure of the projection screen. Then complicated adjustments are required – or the video has to be recomputed to meet the changed conditions.

Fraunhofer FOKUS' newly developed digital dome projection technology uses a so-called autocalibrating display that perform the image distortion correction in real time as the technical conditions in the dome change, thus making the above mentioned adjustments superfluous. Here, a camera and calibration images are used to fine-tune the images to the projection surface. This enables even the tiniest inaccuracies in the distortion correction process to rectified online.

Technology

In principle, the size of the dome which Fraunhofer FOKUS' digital system uses for projection is arbitrarily scalable. The hardware consists of seven commercially available PCs and six standard projectors. These produce six partial images, five on the periphery and one image in the middle of the dome. The image distortion correction is performed in real time by the autocalibrating software developed by Fraunhofer FOKUS. This software also ensures seamless image blending, i.e. that the partial images are perfectly aligned without any visible seams. A dome video of this kind has a maximum possible resolution of 4000 × 4000 pixel and more, which is better than the 2048 × 4096 pixel offered by digital cinema. The projectors are controlled by a PC cluster consisting of a server and six graphics clients. The server not only performs the control, distribution and scene-synchronization functions, it also takes care of system calibration, geometry recognition and parameter setting for the geometry distortion. The actual image predistortion is done for each projected image on the clients in real time. A distributed show player, running in parallel on all computers, is available for generating and playing content. Like a video-editing program, it allows the integration of different components such as graphics, texts, movies and audio – but without the need for the complicated generation of a high-resolution video, such as is required when using a video-editing program. The show player directly synchronizes and plays the various components, making it unnecessary to compute the whole video before projection. 

Application Areas 

The digital dome projection technology can be used whenever the aim is to put viewers in the middle of the action or give them a panoramic view of the objects rendered. The player and autocalibration technology can be used with all different shapes of projection surfaces and combinations of projectors. The technology is suitable for deployment not only in digital cinemas and product presentations but also in planetariums.

Partner

Carl Zeiss