The future of live sport: Heightening the experience for those attending events

The Red Bull Air Race is the fastest motorsport on the planet, attracting the best pilots in the world who demonstrate their skill at manoeuvring around a twisting and turning obstacle course.

The nature of the event means that only one plane is allowed to fly in the sky at any one time. This means spectators get to see a series of time trials, rather than planes going head-to-head, making it hard to judge the performance of one plane over another at live races.

Nevertheless, it is exciting. But those thrills can be enhanced by creating an experience for spectators at the live event that provides a deeper understanding of the complexities of the Red Bull Air Race sport.

That’s where REWIND enters the picture.

“REWIND was founded in 2011 as a high-end CGI and VFX company, but when the Oculus DK1 — a virtual reality headset — was released, the team immediately saw the potential and quickly turned the focus to immersive content creation,” explains Sol Rogers, CEO and company founder.

Sol Rogers, CEO and REWIND founder

“Back then, there were no stitching tools for 360 video and stills, so VFX pipelines with the compositing application, Nuke, were devised to create a quality of content that no other studio was producing. Then the CG artists with a thirst for interactivity, who wanted to push boundaries and had a skill for problem-solving, started looking at the game engine, Unity, and suddenly we were making real-time VR experiences.”

REWIND was selected to apply for the Microsoft HoloLens Agency Readiness Partner Program in 2016. After extensive, hands-on training, the company was given the stamp of approval and chosen as a trusted partner. As part of the Program, REWIND had to deliver a real customer solution, so Rogers chose an existing client, The Red Bull Air Race, and the team created a Proof of Concept (POC), which later became known as Flight Deck.

The future of flight

Rogers took the idea of Flight Deck direct to Red Bull Air Race. “Flight Deck is an experiential marketing exercise to enhance the VIP spectator experience at live Red Bull Air races — giving a heightened user experience and a deeper connection with fans. It’s a Mixed Reality (MR) companion to the Air Race, allowing those at the race events to get, for example, a better understanding of the rules of the sport in an immersive manner.”

The experience, which is shown in the Sky Lounges at Red Bull Air Races, provides the ability to choose either fastest lap of the day or previous laps from a preferred pilot to be overlaid over a miniature race track.

So, exactly how did REWIND go about the task? The key elements in the build were a Hologram explanation of how the race works with visuals and voice over, the facility to learn the rules of one of the fastest air sports in the world, from the Start Gates, to how to take a Chicane, an interactive globe of race stops, a 1:1 scale Edge 540 plane, and the construction of table top model of airplanes. “We also included the ability to watch a faux live race with TV feed and witness two planes fly head-to-head,” says Rogers.

Although there isn’t a competition element to the app, it does have the ability to sync up multiple users.

Assembling the assets

REWIND built the experience using Unity, Touch Designer, Visual Studio and Maya for the assets. The frame rate was key — targeting 60fps was always crucial to this experience, which was quite a challenge because a fast-moving plane caused blurring effects when the speed was too high. To reduce the blur, and keep at 60fps, the developers tweaked the speed of the plane again and again until the best solution was found.

“The beauty of MR is that you can interact with the world around you. With this, we used Touch Designer, which is a software that allowed us to trigger our video feed for the live race via a TV at any point we wanted to in the experience.”

For those less familiar with the technology, Rogers went on to describe the difference between MR and Augmented Reality. “Mixed Reality is a significant advancement of Augmented Reality (AR). Whilst the core premise of both AR and MR is similar — they both add digital elements into the real world — the crucial difference is the underlying technology. Mixed Reality is — for the moment, at least — headset-based, whereas AR is viewed through a flat-screen such as a smartphone or tablet. Mixed Reality is also aware of the geometry of the environment around you – using it as the canvas for you to create immersive content that is defined by the space you are in. Microsoft’s HoloLens is the best example of MR to date, and Magic Leap’s much hyped headset will help further define the category.”

HoloLens allows users to place 3D digital models in the room alongside them. As the Windows-10-based product does not have wires or external cameras, or require a phone or PC connection, users can walk around the objects and interact with them using gestures, gaze and voice.

The use of the MS Gestures built within the HoloLens allowed users to interact with Red Bull planes and a world map. Alongside the gestures, the spatial audio that HoloLens is capable of reproducing, allowed the team to recreate a distant plane flying in, alerting the user to turn around and watch the 1:1 scale of the Edge 540 plane.

Apart from the computer-generated foundations, are there other elements required to create Flight Deck? “It’s mostly computer generated! We were given a 3D model of the plane by Red Bull Air Race, which we reworked into a game-ready asset. We also utilised a live video of the broadcast as a part of the experience, that way, we were able to simulate the feeling of watching a live sporting event, showing users a glimpse of what’s possible.”

Returning to the start

Flight Deck was launched in the VIP lounge at the San Diego race last year, and has since been rolled out to other competitions. It will be demonstrated at the Red Bull Air Race held in Austria in September. Austria is especially important in the history the Red Bull Air Race, as the sport was born in that country in June 2003.

Apart from enjoying the experience for users, does Rogers believe that this type of MR offers a chance to monetise content? “I think there’s some potential here, yes. Selling enhancements to the racing experience on top of the initial layer has the ability to provide an additional revenue stream for the Red Bull Air Race, so I think monetisation could work. It could even take the form of selling subscription packages through the app.”

So, what future plans does REWIND have for Flight Deck? “It gives a deeper understanding of the complexities of the Red Bull Air Race sport therefore giving a heightened user experience and a deeper connection with fans. Using data from the planes to show a race in real-time is something the team is working towards. Other future plans include an AR version for mobiles.”

But beyond air races, can the technology be applied to other sports? “As MR adds a digital overlay to the real world its ability to enhance the viewing of live sport is huge,” sates Rogers. “Furthermore, as it doesn’t transport you elsewhere, like VR does, the social aspect of enjoying a game with friends is still very much in play.

“For example, in the future, football fans could have their experience augmented with player statistics appearing in real-time, the rules of the game and referee decisions could be visualised, while goals could be made more dramatic via colour changes to the pitch and stadium in line with the scoring team’s kit colours.”