Setup for demonstrating interactive binaural synthesis for telepresence applications

In the telepresence research project BEAMING, a prototype system has been set up to demonstrate basic audio and video interaction between two distant locations: the Destination, where 2 Locals are present and the Visitor Site where 1 Visitor is present. This paper describes the auditory parts of this system as well as interfaces to relevant parts of the complete system, including tracking and network streaming.

In the demonstration, the Visitor is wearing headphones and a microphone. At the Destination, the two Locals are both wearing a microphone, while the Visitor is represented using a fixed position Totem with a single loudspeaker. The Position and movements of participants, particularly the head, are tracked and from this sound is rendered to include binaural cues so the Visitor is able to move around in a limited space while perceiving Destination sound as "stationary".

This setup includes 3 main tasks:
Tracking coordinates are combined to calculate directions. This is handled by sharing "global" coordinates across the sites and adding local changes with a low latency, ending with a "direction of sound" for each source. Audio is recorded and transmitted over network. Here bandwidth, latency and transmission reliablity must be adjusted to obtain the best compromise. Bandwidth use and reliablity can be improved at the cost of latency. Finally the binaural synthesis for each source is processed at the listener's site (here Visitor) to have a minimum latency on responding to movement.

The combined system was evaluated by the user experience at the demonstration, with the overall conclusion that interactive binaural synthesis is an important aspect of a fully immersive telepresence application and that we should continue in this direction examining different approaches.