We were very happy to be invited by Stopp LA to join them and Chris Milk in this amazing project - an immersive interactive live performance by Beck!
The interactive experience and the live event was directed by Chris Milk, the interactive visuals was handled by Stopp LA and we at Dinahmoe were responsible for the interactive sound from both a creative and a technical standpoint.
The quite baffling stage and concert area was built in one of Fox Studios sound stages, and the concert was recorded by multiple 360 degree cameras. In the centre of the circular concert area there was a rotating stage were Beck was doing his performance, around the stage was the area where the audience was seated, the audience section rotated in the opposite direction of the stage, outside the audience area there was another section where the supporting musicians were seated, and finally, outside the musicians there was a wall of video screens.
All in all the concert area was about 35m (115 feet) in diameter and the outer section had more than 170 musicians (two choirs, six gamelan bells players, 12 distorted guitar players, a yodeller, a Brazilian rhythm section, a string section, a brass section, and many more..).
The goal for the interactive experience was to get as close to actually being there as possible. The show was recorded with three special cameras from 360 Heroes that captures video in all directions at the same time, placed at three different locations. When the user plays back the video they can choose which camera they want to watch and also move around/rotate/change direction of the view.
To make the audio just as immersive and interactive as the video, Stopp and Chris Milk approached us with a question: "Is it possible to do the same thing as we do with the video with a binaural audio recording*?"
Binaural audio recordings sound very realistic for static recordings, but there hasn't been any good way to control them interactively without using software processing to simulate binaural sound, which of course wasn't an option for this project.
To solve this we got in contact with binaural audio expert Jeffrey Andersson at 3Dio. Together with Jeffrey we designed a recording solution that would allow us to record binaural audio in full 360. With these recordings we were then able develop a solution that lets the user rotate the hearing field in the same way as you can rotate the field of vision.
Dinahmoe did all development for sound including the engine for the playback of all recordings, we also provided stable time codes for the videos to sync to in the interactive experience. Our creative director, Johan Belin, was also on site for the recording in LA to support the audio team and help assure decisions were made with the interactive audio in mind.
* Binaural audio is a very realistic type of stereo recording normally achieved by placing microphones in a synthetic human head.