Jun 30, 2020 9:00 AM

Pokemon Go and Sleep No More Creators Are Teaming Up on AR

Niantic and Punchdrunk plan to turn people into “the hero of their own living game.”

A defining feature of augmented reality is how much the technology's promise has outpaced its real-world use. Despite years of glossy demos, AR’s predominant consumer experiences are still Pokémon Go and an iOS app that measures things. Advancing the medium will take not only impactful hardware—like Apple’s U1 chip and loudly rumored glasses—but innovative games and experiences. The kind that Punchdrunk, the theater company behind immersive shows like Sleep No More, knows better than anyone.

Now Punchdrunk and Niantic, the developer of Pokémon Go and Harry Potter: Wizards Unite, are announcing a collaboration that will span multiple projects. Details such as what the debut effort will look like, when it will launch, and what it might cost remain a closely guarded secret, but as far as crossover episodes go, the cast of characters looks encouraging.

Punchdrunk is no stranger to immersion. Sleep No More, the group’s signature piece, invites audience members to walk through a series of rooms at their own pace, in whatever order they like. The actors perform hour-long loops; as you wander, you cobble together their story fragments into a whole.

The trick of Sleep No More is to give the viewer a sense of free will while maintaining an environment that does not respond to their choices; immersive here does not equal interactive. The performance is fixed, regardless of what the audience does.

“Although it feels spontaneous, everything is meticulously choreographed,” says Punchdrunk founder and artistic director Felix Barrett (no relation). “Our process even at the most transitional beats is completely set. With that sort of precision, and the ability to set and rehearse different scenarios … It’s those sorts of learnings, almost like binary coding in a way, that enable us to think about how one could use that information.”

It’s no stretch to see how that philosophy might extend to AR. Punchdrunk has already dabbled in virtual reality, with a project for the Samsung Gear VR called #believeyoureyes, as well as mobile gaming with Silverpoint, an app that directed players around the streets of London.

“Quite early on I became fascinated in the potential for game mechanics,” Barrett says. “We’ve scratched the surface with a few projects, but I always knew that for an audience to have real agency over the landscapes they traverse, if they could actually grant them that, they could become the hero of their own living game, their own movie. It felt like that was the next hurdle to cross.”

Working with Niantic gives Punchdrunk a leg up. The company has spent years mapping a digital layer on top of most parts of the world, ever since the early days of Ingress, its first game. This rich AR infrastructure is the engine driving its subsequent hits, Pokémon Go and Wizards Unite, and it's what Punchdrunk will tap into for their collaboration, whatever form that takes.

“There’s so many beautiful and interesting and inspiring things out in the world, and thrilling and mysterious things, that we’re programmed to ignore,” says Niantic’s CEO John Hanke. “The goal of a project like this, in my opinion, is to add those points of interaction and that layer of story and gameplay that really causes you to pay attention all the time. It’s not the very particular interactions that this thing is going to deliver, it’s that state of having your eyes open.”

The math here does not quite seem to add up: Can you really make people more in tune with their surroundings by funneling their experiences through a phone? Again, hard to say, given how few details are currently available about the collaboration. But Hanke describes an experience that does not depend on keeping your eyes locked on a screen.

“Imagine that you are in a city with a destination that you’re seeking out and a clue that’s going to get you there,” he says. “You’re looking for that in real life. Your phone is in your pocket, and you have a quest that you’re going on, and you’re totally alive in terms of your senses. When you arrive at a specific location, you may pull out your phone, you may scan something, you may have a very specific interaction with the device, but that’s a point in time.”

Which does sound lovely, at least until you remember that the world is engulfed in a pandemic, perhaps not the ideal conditions for roaming the city streets. Niantic has already made concessions to Covid-19, altering Pokémon Go's mechanics this spring to better enable indoor play. Its collaboration with Punchdrunk started before the novel coronavirus hit, and the companies have had to adjust on the fly.

“We’re taking it hugely seriously and trying to work out new ways to navigate it so that we can still tell our stories,” says Barrett. “Although the planning began before Covid and we’re having to change things, we’re quite adaptive, because we have an audience who is mobile. We’re shifting some things, but we’re not actually changing too much.”

Working with Punchdrunk is just one of several avenues Niantic is exploring to broaden its world beyond Bulbasaurs and broomsticks. Hanke says the company has around 10 new projects in the works, including a previously announced plan to bring the beloved board game Catan into the realm of augmented reality. The company also continues to invest heavily in the Niantic Real World Platform, which lets outside developers tap into its AR Cloud.

A lot has to happen for AR to fulfill its mainstream designs, and to extend beyond the occasional viral hit. Some of those steps—the availability of appealing AR glasses, the abatement of social distancing—are outside of Niantic’s control. A partnership with Punchdrunk, however cloaked in secrecy, at least promises something genuinely new. That’s as good a foundation for the next chapter as any.