Netflix will re-encode its entire catalogue in 2016 to save bandwidth
Tue 15 Dec 2015
After spending four years on the project, video-streaming behemoth Netflix is set to roll out a new video-encoding system which promises a significant cut in general internet traffic, and which will involve the re-encoding of the company’s petabyte of streaming data.
The new system will encode from the raw source material more intelligently, considering whether or not the material itself can really benefit from higher bit-rates, or whether identical quality can be maintained with less space and bandwidth.
Speaking to Variety, Netflix video algorithms manager Anne Aaron explained “You shouldn’t allocate the same amount of bits for My Little Pony as for The Avengers.” This is because animated material often contains large blocks of colour and features relatively little movement, depending on the quality and style of the show, and very large sections can often be re-used in the encoding process, saving space both on the server and in transit.
Conversely high-budget blockbusters tend to have large numbers of scenes with high-levels of activity and change in the frame; in a shot which features a building falling into a river filmed, hand-held, from a helicopter, there isn’t one pixel left that can be re-used, and such material continues to need the maximum amount of data. One of the biggest criticisms during the transition from analogue to digital terrestrial broadcasting was the blockiness that becomes evident when a network devotes inadequate bandwidth to a movie, with artefacts becoming very evident if the camera moves too fast or records action which moves too fast.
The ultimate challenge for encoding – and playback – is a turbulent, moving sea, where every single part of the frame is in motion. Undersea scenes, scenes in fog or extensive use of soft filters all produce dense gradients which suffer very badly when encoded economically, making films such as Das Boot, Superman and The Perfect Storm notorious challenges for encoding enthusiasts.
But Netflix have been keen to make an estimated 20% saving without sacrificing quality of image, and to this end for several months have offered a bottle of champagne to any employees at the company’s Los Gatos offices who are able to tell the difference between two side-by-side playbacks of identical content encoded with both the old and the new method. Apparently no-one has won yet.
Netflix intends to have completed the re-cast of all its video streaming content by the end of 2016, by utilising its vast global array of servers to process chunks of new encoding from splits of its content, before re-assembly – all during the off-peak hours for net traffic.
With an estimated streaming subscriber base of over 60 million, each using an average of 45gb of bandwidth monthly, Netflix pushed approximately 329,400,000,000 gigabytes of data through the increasingly-protesting network infrastructure in 2015, with no let-up in sight for the coming year.
The initial batch of 1000 re-encoded videos are to be completed by the end of March next year, according to the company, with the entire petabyte of video finished nine months later.
The encoding batches are processor-intensive, and take a significant toll on the Amazon compute instances which the company rents for the purpose, though its association with AWS is far more intense in terms of data throughput than actual compute time.
Netflix video accounts for 37% of all North American traffic – the company’s own contribution to the estimated 69% of all web traffic which is occupied by streaming video providers such as itself, Amazon Prime and Hulu. Since future requirements for Netflix streaming may include 4K, higher frame rates and HDR, any savings that it can make with the new encoding system, however impressive, are likely to be consumed within a year or so.
