by Sriram Rao

The Netflix TV app plays videos not only when users choose to watch a title, but also while browsing to find something great to enjoy. When we rewrote our TV user interface in React, we set out to improve the developer experience of integrating video playback into the UI so we could more rapidly experiment with various video-centric user experiences.

A key piece was to expose a declarative interface to the underlying imperative video playback API that is more expressive, easier to extend and better hides the complexity of interacting with a stateful system thus fitting more elegantly into our React application.

A common method of playing video in React involves rendering the <video> element and then using a reference to invoke imperative actions. A good example of this is skipping to a different location within the current video. We don’t actually have a <video> element available to our UI, just an imperative video player API in the Netflix device platform. We decided to map this imperative/stateful API onto a fully declarative/stateless React component because it fit into React better as JSX is declarative, made it possible to hold the state of video in application state, helped avoid the use of refs, and enabled easier layering of functionality via higher-order components (for example, to add text-to-speech).

Destination versus Route

So how did we get rid of the imperative actions? Instead of orchestrating imperative function calls, we send props to our video component that describe the desired target state of video playback. Our video component detects changes in props and decides what video player API functions to call.

Our video component signals when there are changes in video state (e.g., when transitioning from loading to playing) and changes in playback position. Handlers for these changes are passed down in props.

Hello Video!

A component that wraps an imperative API with a declarative facade should be capable of translating changes in input into the correct sequence of imperative API calls.

This is how you can tell our video component to start playback in JSX:

<Video
    playbackState="playing"
    src={videoUrl}
    onVideoStateChanged={this.handleVideoStateChanged}
    onReportedTimeChanged={this.handleTimeChanged}
/>

The playbackState prop is the desired final state. The onVideoStateChanged and onReportedTimeChanged props are handlers for our video component to signal changes in state of video and playback location respectively.

The requested video plays irrespective of the current state of playback. If the video is not already playing, it starts. If the video is paused, it un-pauses. Our video component invokes the right set of imperative API calls to achieve this.

To pause or stop video, the caller sets the playbackState prop to "paused" or "stopped".

Video state changes

Our video component communicates the state of video playback (like loading, playing, paused, etc) so that surrounding UI elements can adapt to or reflect that state.

We found that a component fits well into declarative UI if it:

• hides complexity and variance in the underlying imperative subsystem by providing consistent states and transitions
• converts changes in the underlying system to states that make sense to the UI

An example of the latter is that a switch from one playing video to another will trigger the following state changes in the Video Player:

But the loading of the new video is more relevant to our UI, so it can react by showing a loading indicator. Our video component will smooth this out to:

Extending a declarative API

In most cases extending the API is matter of introducing additional properties. Here’s what it looks like when extended for an audio track, a subtitle track and volume.

<Video
    playbackState="playing"
    src={videoUrl}
    audioTrack={audioTrackId}
    subtitleTrack={subtitleTrackId}
    volume={volumeLevel}
    onVideoStateChanged={this.handleVideoStateChanged}
    onReportedTimeChanged={this.handleTimeChanged}
/>

Modeling dynamic attributes

Extending the API to describe the target value of a constantly changing attribute of the system isn’t straightforward. Such attributes require two properties to describe it.

For example, a single time property isn’t enough because it can become stale very quickly. After video starts at the position specified by the time property, the actual playback location and the time property drift apart. There is also no way to make the video seek back to the position specified by the time property because the value of this property won’t change.

Our solution is to use two time properties — time and reportedTime — where reportedTime serves as a proxy for the actual playback location and helps tackle the drift between the two. You can think of time as the setter and reportedTime as the getter (but one that is delivered periodically via an event). During playback, the UI sets time to the reportedTime it gets so both values stay the same. When the UI wants to skip to a different location it sets time to the target time. Changes in desired position can be detected because time will not match reportedTime.

Here’s an illustration of how this works. Say our video component is rendered with every update to reportedTime (which by the way is not a requirement). Since time and reportedTime advance together it is effectively a no-op. When time differs from reportedTime, a skip is initiated, and no more time updates are sent by our video component. Any renders during the skip are also effectively no-ops if these props don’t change. After the skip completes, time updates resume causing time and reportedTime to be in sync again.

Simpler mental model

The UI can stop tracking the current state and instead focus on describing the target state. Our video component always gets the full and latest target state in props each time which means:

• the UI doesn’t have to throttle, i.e., it doesn’t have to wait for one change to be applied before specifying more changes to the target state.
• our video component only ever has to queue up the latest target state while it is applying a previous change in target state.

The UI also doesn’t have to care about order of operations. It can change multiple properties and expect that the they will get applied in the right order, for instance when audio track is changed and playback state is changed from “paused” to “playing”.

A fully declarative API isn’t without drawbacks. Property bloat is one to watch out for. Modeling of dynamic attributes can be non-intuitive and can result in additional render cycles caused by an increase in state update frequency (like the reportedTime change event would). Our experience is that these aren’t roadblocks and haven’t stopped us from using our video component even on low-end devices.

Declarative across the React boundary

Creating a declarative facade over an imperative API in the form of a React component makes integration into a UI easier. But this can also mean that the declarative facade might have to account for variance in platform / device level support of features and plan for fallback experiences. If you own the imperative API, consider turning that imperative API into a declarative one.

We did exactly that by replacing our video player’s imperative API with a single setTargetState method that takes the same props that our video component does. This has allowed our video component to treat our video player like a react component, passing the props to it and letting it detect and react to changes in the props. It has facilitated better separation of concerns, both between pieces of software and between teams. The video player is in a much better position to decide the actions needed to get from the current state to the desired target state because it has access to the internal state of the player.

There are alternatives even if you don’t own the imperative API. For instance you could encapsulate this variance outside React boundary, in a layer between the declarative component and the actual imperative API.

Better user and developer experiences

As we build new UI concepts for the TV experience we need to solve for tough UI challenges. Our declarative video component has made richer video integrations into our TV UI easier and faster.

Do you want to help us invent the future user interface for the living room? Join us if this sparks your curiosity.