Mixed Reality Creates a TMNT Experience for a New Generation | V by Viacom
Creativity

Mixed Reality Creates a TMNT Experience for a New Generation

A virtual press junket immerses reporters in the new iteration of the Teenage Mutant Ninja Turtles.

Jul 18, 2018

BURBANK, Calif. — When I interviewed Donnie and Mikey—stars of the upcoming 2D animated series Rise of the Teenage Mutant Ninja Turtles, who may be best known as Donatello and Michelangelo to fans of the classic series—it was hard to know where to look. I was enveloped in a virtual Manhattan, surrounded on all sides by skyscrapers and shadows. As they answered my questions, a sewer rat darted across the street.

I was standing in an 8ft x 8ft mixed-reality green-screen cube, wearing a virtual reality headset that enabled me to feel like I was part of their cartoon world. When I asked whether kids or parents were more excited for the return of the Turtles, Mikey looked up as if deep in thought. “Dads are super excited for the show,” he said.

The experience was developed by Nickelodeon’s Entertainment Lab for reporters at Comic-Con. The Lab was tasked with creating a unique way to introduce the press to the new Nickelodeon series, which begins September 17. The result invites reporters to become part of the turtles’ world as they interview 2-D animated renditions of Donnie and Mikey, who are voiced in real-time by the series actors Josh Brener and Brandon Mychal Smith. 

“VR has become integral to how brands communicate their messages and make an impact. Research shows that the way consumers experience brands influences how they feel about them.”

Yes, the virtual press junket is a way to stand out at Comic-Con, an event full of stunts and eye-popping activations. But VR has become integral to how brands communicate their messages and make an impact. Research shows that the way consumers experience brands influences how they feel about them. So what better way to impact reporters whose pens have the power to influence a company’s narrative then having them experience the turtles’ world?

“There is an inherent power in putting users at the center of the action, in letting them experience what’s happening on their own terms,” Colin White, who heads up the tech side of Interbrand’s VR initiative, said in a company article. “It leads to more engagement, even empathy.”

This is the first time the Lab has combined so many elements into one project. There’s virtual reality, live compositing (combining visual elements from separate sources into a single image, to create the illusion that all of those elements are parts of the same scene), recording, puppeteering, virtual cinema and real-time animation.

“This is at the heart of what the Lab is set up to do,” says Chris Young, senior vice president, Nickelodeon Entertainment Lab. “It’s a unique design that offers press an immersive look inside the ‘Rise of the Turtles’ universe and gives them the opportunity to interact with two of the stars from the show.”   

Young’s team has been planning and building the activation since January and rehearsing for six weeks leading up to Comic-Con. In the early days they were creating the technology as ideas came up. Other days they hung the green screen on the rigging using safety pins.

While reporters partake in the front-of-house experience, engineers, technicians, animators and voice actors sit behind a curtain to bring the experience to life. The actors, as guides in the virtual experience, are able to watch the reporters via computer screen so they can direct reporters toward various things in their world. For example, during my interview, Donnie suggested I look down into a puddle. When I did, I saw the reflection of myself as Sandy Cheeks, of SpongeBob fame. (To keep with the tone of the new series, its writers created a playbook of suggested sayings and phrases for the voice actors.)

“It’s a unique design that offers press an immersive look inside the ‘Rise of the Turtles’ universe and gives them the opportunity to interact with two of the stars from the show.” 

Meanwhile, a combination of multiple virtual and a real-world camera record the participant. Live compositing technology combines video of the reporter with the animated turtles into the same scene. As reporters ask questions, a “cartoon DJ,” works a keyboard that controls the turtles’ physical responses in real-time. For example, when I arrive on the set and say “Hi,” the digital puppeteer hits the button to make Donnie wave.

Each reporter has five minutes for their interview. Toward the end of the allotted time, the voice actors gently encourage reporters to wrap up by directing them to watch a billboard featuring a preview of the new series. Each reporter leaves with a flash drive of their conversation. The hope is that they’ll upload the video to run alongside their article.

It takes about five minutes for the engineers to edit and convert the video to a useable file. While that happens, Shazzy Gustafson, a VR and AR engineer at the Lab, introduces the reporter to the voice actors—giving both sides another chance to chat, this time without the virtual cosplay.

And with that, I say goodbye to Donnie and Mikey. Next stop for them: San Diego.