Virtual reality(VR) and augmented reality (AR) seem to be the next generation of visual entertainment. The visual applications are exponential and continue to grow as technology evolves. In terms of visual components, I’m very excited to see what the future holds. Perhaps budget won’t be an issue for travelling around the world and visiting all the hot spots. Extreme sports can literally put you in the front seat and when it comes to communication over continents we may be booking VR visits. The sky is the limit once the technology can be harnessed by society. For now, it is being pioneered by artists and tech junkies.
I was extremely lucky to be invited to the National Film Board for an extraordinary demonstration of a 4 minute VR film entitled, Tidal Traces. I was lead into a room where I was fitted with a headset for my VR experience. When the film was over, I felt like I had participated in it. The experience was personal and interactive. The performers in the film couldn’t see me but I could see them and everything they were responding to. It is an experience like nothing else and will leave you wanting more. Afterwards I was able to talk to the two creators of my VR experience of Tidal Traces.
The 360–video VR dance piece was co-created by Nancy Lee and Emmalena Fredriksson. Nancy contributed her directing and post-production skills and Emmalena choreographed the dance. Using their combined efforts, they have made a contemporary dance film that invites you into the ocean alongside the dancers for the ultimate experience in viewership. You can almost smell the ocean and feel the water.
“ How did you both meet and become collaborators on this incredible project?”
Nancy answers, “We were brought together through a mutual friend that left the project early in. We first got together shortly before May of 2016 to discuss the project. We made a demo reel in July. I built my own VR rig using 6 GoPro’s on a 3 D printed rig. It was a much smaller and simple design with a tripod. We shot the demo film at the same location and using the same dancers. Through that process we were able to gain a lot of insight into the location and the challenges we might encounter shooting the full production.”
Emmalena continues, “Then we got a year long residency at the Dance Centre where we got to play with the material. We did a lot of public presentations, people were coming in to view the demo. We had an open workshop on choreography for Virtual Reality. We were half way through the residency in the winter of 2017 when we were able to pitch it to the NFB. At that point we had the demo and we also had a lot of information on what we wanted to do if we had another chance. Once the NFB said yes, it was a fast turnaround. It happened in Feb. of 2017.
You were in Asia, I was in Europe, and the dancers were also in other engagements. We contacted the NFB to explain that we have these weeks in July in which we can do it and they said, we’ll make it happen. The postproduction took a little longer. In total, from the first NFB meeting, it was 6 months in total to shoot and edit.”
The dancers in the VR film are Zahra Shahab, Rianne Svelnis, Lexi Vajda. They had to film in July because it was too cold in the winter. Because they were also working with dancers that are in pretty high in demand, in terms of scheduling, they all had to be in the country at the same time.
How did you practice the dance moves differently to incorporate the 360 degree camera?”
Emmalena, “We worked in a few different ways. Some of film shows set or fixed phrases but we also worked with a lot of improvisational tasks, meaning, some of the decisions on the moves were from the dancers. We tuned it and decided which kind of frame and how those movement choices could be made. We couldn’t make just any move, they had a specific task to move a certain way, so they had to improvise around that.”
Nancy, “We would go out with my technical director, PA and the equipment into waste deep water and look for the perfect patch of sand to shoot in. Not all of the sand is suitable for dancing. Although it was in July, the dancers would only enter the water once a spot had been chosen and equipment set up because of the cold water. When we called them, it was a 20 minute walk to the 360 camera. We’d turn on the cameras and retreat back and lay in the sand. As a director and choreographer, we had to remain silent and out of site. As an alternative, we’d support them, give them agency, empower them to make creative decisions about their own performance quality.”
When you watch and listen to Tidal Traces, it absorbs you and teleports you to another place and time. When you combine the sound with the visual you feel like you are the fourth dancer. After one more rehearsal, I was sure I had all the moves down. I was surprised to find out the sound was edited in later. The splashes and water droplets came from a recording in Trout Lake. Nancy’s composer and sound designer Kiran Bhumber took splashes from the lake and designed the sound. Kiran studied how the dancers were walking around the water and spatialized the sound accordingly. It is Ambisonic, so the sound is spatialized. Wherever you look, the sound will find you. To capture the images, a GoPro Odyssey camera was utilized.
“Did you also rehearse outside the water?”
Emmalena, “We did both. We first rehearsed in the studio before going out to the site. Rehearsing on site was easier. It was a sunny day, we took lots of breaks, we were able to use higher dynamic movements because the sand conditions were really good, the water conditions were really good but then filming it, we had a lot of different things to think about. We went out to the site and sourced some material that gave us the assurance that we can do this in the water and sand. We then went back to the studio and created a long sequence and felt like we had it all worked out. We then went back out for the technical rehearsal and it was very hard to perform the sequence although all the material had come from the water and the sand. The temperature was different, the sand was different, it was really hard to create a consistency to perform the same movement. It’s the reason we decided to go with more structured improvisations, where the dancers had more multiple choices. For example, if they were turning and falling they had maybe 4 options how to come out of that turn. There was a lot of problem solving both in the dance and the technical part that caused us to be more in the moment and become more responsive to what was happening.”
“Was dance the first idea to apply the VR technology?”
Nancy, “It was our idea and the reason we were able to meet. It was our intersection. I have previous experience creating dance films and an area I’m already interested in developing. For this one specifically, it was like, how do we design and plan a piece that justifies the use of 360 in video. We could’ve made a regular dance video but we wanted to dive deeper into, why this medium, why 360 video? How to compose, how do we plan in terms of film size, the location, the narrative, the choreography, the composition? How do we plan that so it can justify the use of this medium?”
Emmalena, “One of the comments we get the most from people, is that, we come at it from an artistic perspective. With 360 VR technology developing so fast, there’s a lot of interest in how to display it. A lot of projects want to show what the technology can do but the artistic content of the work is secondary. At the NFB they said they want to support your artistic vision first, we don’t have the equipment to display the latest technological advances of VR 360 so we want to make an artistic project that is about bringing a story or an experience forward rather than, this is how cool the technology is. I have a background in contemporary dance, as a performer, choreographer and teacher. I haven’t worked much with film before so this is very exciting for me but I have done a lot of performances in 360. I’ve performed a lot and choreographed for 360 before but it’s very different when it’s live. I can tell when you’re looking at me but in the headset, I don’t know. The unpredictability of the delay is very different from a live performance.”
Nancy, “ Also, different viewers have different viewing behaviors and viewing habits in this environment. From a film perspective, that’s why we chose that site. We could’ve shot somewhere else but it wouldn’t have justified the use of this medium. That’s why we also chose this site. If you’re the viewer you might want to focus on the sand, the sky, the water. We also wanted to offer a unique kind of value.”
Emmalena, “Everyone’s experience will be different from the previous viewer. It changes all the rules according to what you’ll choose to look at or where you’ll move. When you’re in a theatre you can dictate where people are going to look, but because of the composition, we like to say the viewer becomes the fourth dancer because everyone is going to make a different decision. The compositional tools that we’d use in a theatre is out the window because you don’t know what people will be looking at. You have a solo over there, a duet over there, which one do you choose? It became more about choreographing different choices and different experiences rather than seeing the symmetry between them. We made the dance a little more subtle especially in the beginning because we wanted to give people time to observe the water as well as the surroundings. If we had choreographed a super complex sequence, it might be lost to a wandering eye. It would be a waste.”
“Is this your first VR film?”
Nancy, “ It is, I have a filmmaker, interactive interdisciplinary media artist background. Which means, I design interactive interfaces as installations or performance interfaces. For example, I have another project where I designed these swing sets that have sensors on them, so you can sit on the swing and depending on your behaviour and how you swing, how you interact with the swing set, the rotation data from the swing changes the audio/visual experience. We have projections in the whole space with sound, so it’s immersive. I already work in immersive installations, so the 360 media felt right for me as a filmmaker and as an interactive VR artist. It’s cinematic but at the same time have some form of agency to interact in this submersive environment. I’ve been doing interdisciplinary art for the last 5 years.”
Editing and colour correction were done using Adobe Premiere Pro. Rotoscoping was achieved using Mocha VR’s stereoscopic tools along with depth-map mask compositing and nadir/zenith hole patching in After Effects. Working with 8192 x 8192 resolution footage was a data management and processing challenge. Even with 2K proxies, it was impossible to edit with real-time playback, so footage had to undergo long renders before it could be evaluated for timing and pacing.
Spatial audio was designed using Reaper and the Facebook Audio Workstation plugins. Multiple audio formats were generated to accommodate different delivery platforms. The binaural audio was encoded in 1st-order AmbiX as well as 8-channel TBE and 2-channel head-locked stereo.
“Considering so much effort went into the four minute film, do you wish you would’ve made it longer?”
Emmalena, “It was a conscious choice to make it 4 minutes. With VR and 360, cyber sickness becomes a real thing. When the film is too long it gets very disorientating. Also for distributing purposes, because we can’t screen it to hundreds at a time, having a 4 minute film turnaround is easier to wait for a viewing at a festival installation.”
Nancy, “Also with the technical side within the post production process and being able to render it, a 4 minute film was an ideal length. It took so long to render the files. It turned into a 16 Terabytes project for a 4 minute film. If it was any longer it would’ve been difficult in terms of file size. Even when I was working with Vincent McCurley, our creative technologist, if he chooses to transfer the hard drive, it requires 3-8 hours. Given the magnitude of the file size and the computer power that it requires to operate, creating something nice and short is ideal.”
How did you share the creative process?”
Nancy, “We covered different areas. Emmalena worked more in the pre-production, the rehearsals, the planning, the choreography, working with the dancers and the production side. I had the task of piecing it together while maintaining the quality of the video, satisfactory to Emmalena. That is the joy of working together. When Nancy was going through the edit, she’d be looking at the light and composition and I’d be looking at the performance of the dancers. We had to negotiate our two interests. It makes for a richer experience as an artist. We’re both finding that we’re growing and learning.”
“Where was the location?”
Nancy, “It was Boundary Bay and we only had a three hour window to film in the two days because of the tide.”
Emmalena, “When it came time to rehearse in the water, the dance moves were limited by the splashing of the water. Depending on the depth of the water, it would impact the sound. When it came to the shoot we had to use long takes for the purpose of less edits. It became more subtle but it also allowed us a theme to emerge that would invite the viewer into the dance rather than force the dance onto the viewer. When they cross the middle, we’re gonna work on how they use their gaze, how they’re running and what is their relationship to each other? how can their movements in the 360 experience make the viewer turn around?”
“How would you go about displaying it to the public?
Emmalena, “Within the lobby we’ll have private installations set up with a curtain surrounding you for a private experience. There will be a stool for sitting and goggles available.”
Depending on the device you choose to view it on, the resolution and audio can fluctuate. For optimum enjoyment, an 8 K device is the best option.
“On the technical side, did you work with a team?”
Nancy, “I worked with one other person. I did all the editing and then Vincent McCurley who works for the NFB became our creative technologist. He helped with the roto-scoping process. We had to roto-scope out some of the sand bars and replace it with water. That process was pretty intense considering the 8 K format and that its stereo- scopic, so you have to offset the left and right eye depending on the distance of the dancer to the lens.
Kiran Bhumber composed the music. She’s currently at the University of Michigan completing her masters. Because of the facilities there, she was able to utilize them to create the sound. She also worked with 2 instrumentalists for the composition. There is software called facebook 360 audio engine which allows you to spatialize the sound. So essentially, she can assign certain sounds to where the dancer is in the 360 environment. She can track where the dancer is moving so the sound can also move in a spatialized way. Certain movements have certain percussions assigned to it and certain movements have violin plucks assigned to them. The sound also helps signal changes or potential directions you should be looking at depending on where you are in the environment.”
“Would you consider teaming up again on another project?”
Emmalena, “Yes! This project has taken on so many different forms and going on two years, we clearly enjoy working together. I’ve learned a lot about VR and 360 through Nancy and that’s what’s so nice about the collaboration. We come with different skill sets, which makes it more enjoyable to work together.” We wanted to create a certain kind of experience and had a curiosity for what this medium can do for dance and what can dance do for this medium. It was a driving question for us and made it more exciting.”
The 360-video dance experience has been to Berlin, Paris and Tokyo. Its Canadian premiere was at the Women in Film and Television film festival. It will also be showing at a youth festival on April 8, called Reel 2 Reel. For more information, please visit http://www.r2rfestival.org/