Updating Browsers Quickly: Flags, Respins, and Components – text/plain

By this point, most browser enthusiasts know that Chrome has a rapid release cycle, releasing a new stable version of the browser approximately every six weeks. The Edge team intends to adopt that rapid release cadence for our new browser, and we’re already releasing new Edge Dev Channel builds every week.

What might be less obvious is that this six week cadence represents an upper-bound for how long it might take for an important change to make its way to the user.

Background: Staged Rollouts

Chrome uses a staged rollout plan, which means only a small percentage (1%-5%) of users get the new version immediately. If any high-priority problems are flagged by those initial users, the rollout can be paused while the team considers how to best fix the problem. That fix might involve shipping a new build, turning off a feature using the experimentation server, or dynamically updating a component.

Let’s look at each.

Respins

If a serious security or functionality problem is found in the Stable Channel, the development team generates a respin of the release, which is a new build of the browser with the specific issue patched. The major and minor version numbers of the browser stay the same. For instance, on July 15th, Chrome Stable version 75.0.3770.100 was updated to 75.0.3770.142. Users who had already installed the buggy version in the channel are updated automatically, and users who haven’t yet updated to the buggy version will just get the fixed version when the rollout reaches them.

If you’re curious, you can see exactly which versions of Chrome are being delivered from Google’s update servers for each Channel using OmahaProxy.

Field Trial Flags

In some cases, a problem is discovered in a new feature that the team is experimenting with. In these cases, it’s usually easy for the team to simply remotely disable or reconfigure the experiment as needed using the experimental flags. The browser client periodically polls the development team’s servers to get the latest experimental configuration settings. Chrome codenames their experimental system “Finch,” while Microsoft calls ours “ECS” (Experimental Control System”) or “CFR” (Controlled Feature Rollout).

You can see your browser’s current field trial configuration by navigating to

chrome://version/?show-variations-cmd

The hexadecimal Variations list is generally inscrutable, but the Command-line variations section later in the page is often more useful and allows you to better understand what trials are underway. You can even use this list to identify the exact trial causing a particular problem.

Regular readers might remember that I’ve previously written about Chrome’s Field Trials system. You can find some documentation of Edge’s ECS here: https://docs.microsoft.com/en-us/deployedge/edge-configuration-and-experiments. The tl;dr is that you must ensure https://config.edge.skype.com is reachable and you should think long and hard before setting the ExperimentationAndConfigurationServiceControl policy.

Components

In other cases, a problem is found in a part of the browser implemented as a “Component.” Components are much like hidden, built-in extensions that can be silently and automatically updated by the Component Updater.

The primary benefit of components is that they can be updated without an update to Chrome itself, which allows them to have faster (or desynchronized) release cadences, lower bandwidth consumption, and avoids bloat in the (already sizable) Chrome installer. The primary drawback is that they require Chrome to tolerate their absence in a sane way.

To me, the coolest part of components is that not only can they update without downloading a new version of the browser, in some cases users don’t even need to restart their browser to begin using the updated version of a component. As soon as a new version is downloaded, it can “take over” from the prior version.

To see the list of components in the browser, visit

chrome://components

In my Chrome Canary instance, I see the following components:

As you can see, many of these have rather obtuse names, but here’s a quick explanation where I know offhand:

• MEI Preload – Policies for autoplay (see chrome://media-engagement/ )
• Intervention Policy – Controls interventions used on misbehaving web pages
• Third Party Module – Used to exempt accessibility and other components from the Code Integrity protections on the browser’s process that otherwise forbid injection of DLLs.
• Subresource Filter Rules – The EasyList adblock database used by Chrome’s built-in adblocker to remove ads from a webpage when the Safe Browsing service indicates that a site violates the guidelines in the Better Ads Standard.
• Certificate Error Assistant – Helps users understand and recover from certificate errors (e.g. when behind a known WiFi captive portal).
• Software Reporter Tool – Collects data about system configuration / malware.
• CRLSet – List of known-bad certificates (used to replace OCSP/CRL).
• pnacl – Portable Native Client (overdue for removal)
• Chrome Improved Recovery – Unsure, but comments suggest this is related to helping fix broken Google Updater services, etc.
• File Type Policies – Maps a list of file types to a set of policies concerning how they should be downloaded, what warnings should be presented, etc. See below.
• Origin Trials – Used to allow websites to opt-in to experimenting with future web features on their sites. Explainer.
• Adobe Flash Player – The world’s most popular plugin, gradually being phased out; slated for complete removal in late 2020.
• Widevine Content Decryption – A DRM system that permits playback of protected video content.

If you’re using an older Chrome build, you might see:

• Signed Tree Heads – Certificate Transparency tree roots. This component was removed.

If you’re using Edge, you might see:

• Trust Protection Lists – The list of known trackers and organizational lists used by the Tracking Prevention feature. Presently, this list derives from Mozilla’s Content Blocking list, which in turn derives from Disconnect.me‘s list.

If you’re using the Chromium-derived Brave browser, you’ll see that brave://components includes a bunch of extra components, including “Ad Blocker”, “Tor Client”, “PDF Viewer”, “HTTPS Everywhere”, and “Local Data Updater.”

If you’re using Chrome on Android, you might notice that it’s only using three components instead of thirteen; the missing components simply aren’t used (for various reasons) on the mobile platform. As noted in the developer documentation, “The primary drawback [to building a feature using a Component] is that [Components] require Chrome to tolerate their absence in a sane way.“

Case Study: Fast Protection via Component Update

Let’s take a closer look at my favorite component, the File Type Policies component.

When the browser downloads a file, it must make a number of decisions for security reasons. In particular, it needs to know whether the file type is potentially harmful to the user’s device. If the filetype is innocuous (e.g. plaintext), then the file may be downloaded without any prompts. If the type is potentially dangerous, the user should be warned before the download completes, and security features like SafeBrowsing/SmartScreen should scan the completed download for malicious content.

In the past, this sort of “What File Types are Dangerous?” list was hardcoded into various products. If a file type were later found to be dangerous, patching these products with updated threat information required weeks to months.

In contrast, Chrome delivers this file type policy information using the File Type Policies component. The component lets Chrome engineers specify which types are dangerous, which types may be configured to automatically open, which types are archives that contain other files that may require scanning, and so on.

How does this work in the real world? Here’s an example.

Around a year ago, it was discovered that files with the .SettingsContent-ms file extension could be used to compromise the security of the user’s device. Browsers previously didn’t take any special care when such files were downloaded, and most users had no idea what the files were or what would happen if they were opened. Everyone was caught flat-footed.

In less than a day after this threat came to light, a Chrome engineer simply updated a single file to mark the settings-content.ms file type as potentially dangerous. The change was picked up by the component builder, and Chrome users across all versions and channels were protected as their browser automatically pulled down the updated component in the background.

Ever faster!

-Eric

Appendix A: Component Update Policy

This information is current as of October 2020 but will likely change over time.

The Microsoft Edge ComponentUpdatesEnabled policy can be set to default (unset), enabled, or disabled.  In the case that a component supports the policy and the policy is disabled, the update will be blocked.  Otherwise, the component will be updated when a new version is found.

Component NameCan be disabled by policyApproximate size of CRXAdobe FlashYes13MBSubresource Filter RulesYes77kBTrust Protection ListsYes54kBEdge Improved RecoveryYesNot in useCRLSetNo22kBOrigin TrialsNoLess than 1kBWidevine Content Decryption ModuleYes5MB

Currently, only CRLSet and Origin Trials forcibly download their updates.

The update check step is a request asking what updates exist, and is very small (>1kB typically).  The CRLSet and Origin Trials will download based upon that information either way, but they are also fairly small.

The update check also allows the browser to know if components like Flash or Widevine are too old to use safely.  Widevine ships with a version bundled into the browser, and updates every few months otherwise; leaving updates disabled for it could impact viewing of some DRM protected videos until the next browser release.

Flash updates monthly, but is only sent to users that already have it or who request it (in edge://settings/content/flash).  The Flash configuration can additionally be disabled by Flash/Plugin specific group policies; see DefaultPluginsSetting – and set that to Block.  Then Flash will not update, even if other component updates are enabled.

The update check runs every five hours; if configured for fast update, it starts 10 seconds after browser launch, but more normally 5 minutes after browser launch.  (Extensions do update after 10 seconds though, and they likely look similar from a network perspective.)

The main Edge installer carries the then-current version of each component, so disabling ComponentUpdates but regularly installing all browser updates should keep each component no more than a few weeks out of date.