Lessons from The Super Bowl in streaming sports at scale

‍

There’s nothing like live sports. And the same can be said of live sport streaming. The streaming rights to huge sporting events command top dollar because the levels of fan engagement are unrivaled and that passion runs from top to bottom, from players on the pitch to content providers and networks and finally to fans in stadiums and on sofas. The live aspect is vital to the global appeal of events like the Super Bowl. It’s the responsibility of streaming networks to create that same collective experience for their users wherever they are. That means high quality live video needs to be hitting their screen as fast as possible so that they can feel like they are watching together with every other fan. Lose that and you lose its appeal. 

‍

When an event is watched by many millions of people, streamers need to harness the power of the greatest technology and intelligent systems to get it to them and deliver that primal experience of seeing your team battle it out with the best. 

‍

The next Super Bowl is going to be hosted by Fox. While it isn’t going to be on till next year, preparations are already well underway and the network will take make use of the other marquee events throughout the year to get their service Super Bowl ready through what they have called their “Season of Champions” which includes the 2022 MLB season & World Series, the 2022 FIFA Men’s World Cup, the 2022 NFL season & NFC Championship Game, the 2022 Midterm Elections, the 2023 Super Bowl LVII and ends with the 2023 FIFA Women’s World Cup, an event that, if this year’s Euros are anything to go by, is set to be bigger than ever.

‍

This year, the NFL NFL announced on Tuesday that an estimated 208 million-plus viewers watched Super Bowl LVI. Each year, the proportion of that which is streamed rises each year, last year it was around 3%, this year it crossed the 6% mark and Fox are hoping to be able to reach 10% in 2023. This not only shows how much the technology is improving but how much further it has to go. 

‍

Broadcast and digital 

‍

One big shift from the last time the network hosted the games is their decision to bring the broadcast and digital teams under the same roof. In the words of the SVP of Video Platforms for Fox Corporation Steven Thorpe, this will mean that there are “more eyes on glass to make sure that it is flawless”. 

‍

While considerations may vary and many developments in digital streaming have occurred thanks to the industry embracing the differences between digital and broadcast, the decisioning and monitoring can actually mirror and complement each other. Their objective, after all, is very much the same: to get users their content at the highest possible quality.

‍

Prepare for the unexpected 

‍

One lesson Fox took from the last time they hosted the Super Bowl is the importance for time, testing and interactions. While the company did not run into any major issues the last time they hosted the Super Bowl, Steven Thorpe explained at NAB Las Vegas that “the timelines got a little compressed” in the run up to the event. And this is after they started a year prior. This year preparation for the 2023 game started months before the 2022 game had even been played. This time around, he says, their goal is “this summer to actually wrap up development by mid to late summer so that we can really spend the whole fall just dialing everything in.”

‍

As Thorpe explains, “it really is the unexpected that takes up so much of our focus.” This means an investment in a huge amount of redundancy to ensure that the user experience is protected as they procure a measurable part of the internet to deliver the event and need to be able to factor in everything from a solar flare to a fire in their facility. Their scenarios list that runs through each individual piece of infrastructure and transmission path to make it is fault tolerant is extensive. They will also run chaos testing from six months before the event where they rehearse the stream as a team intentionally set about wrecking the system so they can try and experience every detail of what’s wrong and, most importantly, mitigate customer impact. 

‍

Thorpe still admits, however, that unexpected issues arise during the event but having their workflows developed and established with collaborators such as Amazon, encoders, CDNs and other teams, is invaluable. Fortunately for him and his team, his network also hosts numerous other massive events to put his work to the test beforehand. 

‍

Observability 

‍

One major advantage to digital is that it gives the possibility to understand the user journey at an even greater granularity. This means they don’t have to wait for the Nielson report the next day to reflect on how they can do things better but can react immediately or build systems that can react for them and more quickly. 

‍

This observability is vital to the Fox mission and it allows them to view all the way from the broadcast center through the transmission systems and the workflow down to the actual end user device. They can see if an encoder is having an issue or if it is a problem with the CDN peering partner and so if a problem crops up, they can mitigate and route around it in real time. 

‍

This means they can understand the customer journey all the way through the stack and understand what the customer is experiencing from errors, to picture quality shifts, to rebuffing rates and more and then go back up the chain to see where anything may be going wrong and fix it before the customer even notices, keeping their eyeballs on the show.

‍

EXTRA

‍

The executive engineer at Fox, Mayur Srinivasan also explains that they built a highly redundant transmission pipeline last time in 2020 that meant each incoming contribution signal was an RTP-FEC feed with video at 720p60 20 Mbps. In 2023, they are hoping to be able to exploit the state of the art infrastructure from their broadcast facility to deliver 1080p60 HDR source. They are hoping this will help resolve any transmission issues which is an issue which requires them to answer many more questions such as: whether they should upgrade their codec format to HEVC for the contribution feed and what the optimal bitrate would be if they choose to do so or how they minimize transmit hops on the contribution feed before they hit geo-redundant cloud encoders in multiple cloud regions.

‍