tl;dr

Containers is an innovative and unique Firefox feature. The reception has been overwhelmingly positive, and so we used a Test Pilot experiment to refine our user experience. We have graduated Containers from Test Pilot to addons.mozilla.org to make it available to all Firefox users!

This post provides details of our experience in Test Pilot. We learned:

• People make heavy use of Container tabs, customize their Containers, and use Containers & tabs in a variety of ways
• Key bug-fixes and enhancement areas for Containers
• Which underlying contextualIdentities APIs will be needed most for add-on developers to make Containers add-ons covering many more use-cases

A brief history of Containers

We all do lots of different things online: work, shopping, research, watching too many YouTube videos etc. etc. Containers is an experimental feature that lets users separate their browsing into distinct contexts within Firefox by isolating things like cookies, local storage, and caches into discrete buckets. In practical terms this system has several positive benefits:

• Online privacy: Online ads and cookies cannot follow users from one Container to the next.
• Account Management: Multi-account users can stay logged in to multiple account instances at the same time.
• Organization: For heavy tab users, Containers add a layer of visual organization to the Firefox interface.

Containers began as an experiment in Firefox Nightly, and received extremely positive notice from tech press. From the start, we believed the feature showed great potential, but suffered from a number of underlying usability issues. Missing features and easily accessible entry points made Containers clunky to use. More significantly, it wasn’t clear how to explain the feature to mainstream users. With so many potential use-cases, which one would resonate most with users? What were the big user problems that Containers solved? How could we introduce users to such a novel user interface?

So we went to work, gathering feedback from our Nightly users to determine what features were lacking from Containers. In addition, we conducted a design sprint to revisit underlying assumptions about contextual identity online. The prototypes that emerged from this sprint laid the foundation for the presentation and UX of the eventual Test Pilot experiment.

Early prototype from our design sprint

Here’s how it went down

In the transition from Nightly to Test Pilot, we left the underlying technology behind containers intact but made numerous changes to the presentation and user experience of the feature. Most notably, we did the following:

We moved the UI for Containers from the preferences into an icon in the browser’s toolbar.

We added an onboarding flow for first time users reiterating the value proposition(s) of the feature.

We added a Container on-boarding experience for the add-on.

We also enhanced the UI for users to manage their Containers and tabs, and introduced features to show, hide, and move a container’s tabs.

We created a new interface for editing Containers.

We added a Container detail panel to hide, show, move, and manage its tabs.

To increase discover-ability and kick-start new users’ habits into opening container tabs, we also changed the tab bar behavior to show available containers as soon as the user hovered over the plus button.

Hovering over the tab-bar “+” button shows available Containers for a new tab.

For this experiment, we collected data to learn:

• Do users install and run this?
• How do users create container tabs?
• Do users customize their containers?
• (How) do users manage their container tabs?
• and more

Here’s what we learned

Do users install and run this?

We hypothesized that Containers would be a popular feature for power-users. As expected, our overall installation number of ~10,000 users was lower than most other Test Pilot experiments. However, since the Containers feature is part of users’ critical path of actions (opening tabs), the daily active user ratio for Containers was higher than all other experiments.

(Note: The engagement data had already been truncated by the time of this post, so we’re unable to provide specific engagement ratio numbers here. But it was high!)

How do users create container tabs?

Our tab bar hover UI certainly led people to use it the most.. In an early month of the experiment, 228K container tabs were opened via the tab bar — nearly 10x more than the next most popular source: the pop-up panel.

“tab-bar” was the source for nearly 10x as many container tabs as the panel pop-up

However, we quickly received a topic on our Mozilla Discourse and an issue in our GitHub project (again — Containers users tend to be power-users) that the hover UI was too intrusive. While we agreed that we would probably not “ship” Containers with hover UI, we decided to keep it for a while to keep reminding new users about their Containers.

Do users customize their Containers?

We also measured how many times users clicked to add, edit, or delete Containers.

“edit-container” and “add-container” were clicked approx. 3 times per user

On average, “edit container” and “add container” were clicked approximately 3 times per user, suggesting that users do indeed customize their containers. But the clearest signal that users customize their Container experience is the number of tabs opened in each Container.

We shipped the experiment with 4 Containers: “Personal”, “Work”, “Finance”, and “Shopping” with User Context IDs 1–4. And while Personal (1) and Work (2) were the most common user context IDs for tabs, the next most popular user context IDs for tabs were 6 and 7 — custom containers!

Note: to protect users from revealing potentially personal information, we never recorded container names — only the context IDs

This told us that users indeed make their own Containers and open many tabs in their custom Containers!

(How) do users manage their container tabs?

In this experiment, we added the ability to hide, show, and sort Container tabs. So, we wanted to learn how many people would use those features.

Of our experiment users, 13% clicked to sort their container tabs, 3.4% clicked to hide a container’s tabs, and just 1.5% clicked to move a container’s tabs to a new window.

Building, Measuring, and Learning in Test Pilot

Containers users turned out to be tech-savvy and passionate. Many of them filed issues in our GitHub project for bugs and enhancements. We embraced this early in the experiment: we encouraged users to up-vote issues and we prioritized our changes by the most popular issues. These issues turned into important bug-fixes and great product enhancements.

New Feature: “Site Assignment”

From very early on, the most popular feature request was to assign sites to automatically open into a certain Container. It was first filed on March 2. In 2 weeks, Jonathan Kingston (the #1 contributor on this experiment!) had built a work-in-progress version of the feature, and in another 2 weeks we had refined it, and instrumented it with additional telemetry to measure its effects.

We added the ability to assign sites to always open into a certain Container

To prevent accidental container tabs (and potential data leakage), we provide a confirmation dialog.

So, in Test Pilot, we were able to build, deploy, measure, and learn about a new feature in 1 month.

And its effects were dramatic. 9% of all users added at least 1 container assignment.

And those assignments resulted in 127k more container tabs!

In fact, as the experiment went on, the site assignment feature became the #1 most common source for container tabs — even more than the tab bar.

Site assignment became the #1 source for users opening Container tabs

This is a great improvement, because the more people use container tabs, the more their online activity is separated and protected from various kinds of tracking and hacking! The ability to build, measure, and learn during the course of a Test Pilot experiment is quite powerful.

New Bug-fixes

In addition to feature requests, we received many bug reports in our GitHub issues. While many of these were bugs in the add-on code, some of them were actually bugs in the underlying Firefox platform.

Over the course of the Test Pilot experiment, 36 underlying bugzilla bugs were resolved. The Test Pilot audience brought a fresh perspective to Containers that helped us to prioritize and fix some of the lower-level Firefox bugs. So, our Test Pilot work also contributed to the core Firefox feature.

New Features Forever: contextualIdentities API for developers

After adding some new features, it became obvious that we would not be able to solve every use-case for Containers. The architecture of Containers is designed first and foremost as a security and privacy technique. The effects for online accounts and tab management are not always on the top of the minds of Containers engineers. And many of the feature & change requests for tab management actually conflict: different people use Containers in different ways.

So, while experimenting with the front-end UI for Containers, the team also built out the contextualIdentities API for Firefox add-on developers. In fact, over its development lifecycle, the Test Pilot add-on used different libraries — Bootstrap, SDK, and Embedded WebExtension code — to maintain Firefox compatibility as the underlying contextualIdentities API matured. (Watch one of those videos of people changing the tires of a car while it’s moving and you’ll have an understanding of what it was like.)

Because the contextualIdentities API is available to developers, we are encouraging users to upvote enhancement issues, and encouraging add-on authors to look at those most popular enhancements as ideas for their own add-ons. There’s already a growing number of additional Containers add-ons, and Jonathan has started to curate them into a collection on addons.mozilla.org (AMO).

Here’s what happens next

Speaking of AMO … we are graduating the Containers add-on from Test Pilot to AMO. Our Test Pilot experiment demonstrated that Containers is a unique, innovative, and powerful feature for Firefox users. We also learned that — besides a few quick changes to make Containers more effective for everyone — different users have different use-cases for Containers.

So we’ll maintain the core experience— we will triage new issues as they are filed, and work on the most up-voted issues for our future releases. We’ll enhance the contextualIdentities API, and support and encourage add-on developers to create a wide array of Containers add-ons.

Thank you!

Thank you to all the Test Pilots who installed Containers, provided feedback, and filed issues for bugs and enhancements! We are very happy with the refined Containers experience we were able to launch on addons.mozilla.org.

Want to try a new experiment? Visit https://testpilot.firefox.com.

Luke Crouch, Privacy & Security Engineer

Tanvi Vyas, User Security & Privacy Tech Lead

Jonathan Kingston, Privacy & Security Engineer

John Gruen, Test Pilot Product Manager