Relative Performance of Rich Media Content across Browsers and Operating Systems

Relative Performance of Rich Media Content across Browsers and Operating Systems

Monday, March 1, 2010

Two of the things that Flash is often criticized for is that it:

1. Uses too much CPU
2. Performs significantly worse on the Mac than on Windows

This got me thinking about whether some quick tests would bear this out, and if so, whether it was isolated to just Flash content.

Below are some raw numbers showing CPU usage of various rich content across different browsers and operating systems. This includes video deployed via HTML 5, JavaScript / Canvas animations, Flash Video, and Flash animations. This is by no means a scientific study, but I do think the results show that:

1. Flash does not necessarily perform worse on Mac as opposed to PC
2. All rich media content, including that created with JavaScript / HTML 5 content show (sometimes widely) varying levels of performance across browsers and operating systems

All numbers below show CPU usage as a total of all CPU resources available on the system. This means that if you are running on a machine with more than one core processor, then CPU usage can be over 100%. This is how Mac shows CPU usage, and I feel it gives a clearer picture of how much CPU any individual item is using. Windows CPU usage have been normalized to show CPU usage in terms of this overall CPU usage.

All tests were run on the following hardware:

Model Identifier: MacPro3,1 Processor Name: Quad-Core Intel Xeon Processor Speed: 3 GHz Number of Processors: 2 Total Number of Cores: 8 L2 Cache: 12 MB Memory: 8 GB Bus Speed: 1.6 GHz

I have both a Windows 7 and OS X partition on this computer, and Windows 7 tests were run while booted into Windows 7.

For browsers, I used the latest release browser versions. For Flash Player, I used the latest labs release (Flash Player 10.1 Beta 3 : 10,1,51,95).

Mac OS X Versions

Mac OS X 10.6.2 Google Chrome 5.0.307.9 beta Safari 4.0.4 (6531.21.10) Firefox 3.6 Flash Player MAC 10,1,51,95

Windows 7 Versions

Windows 7 7600 Google Chrome 4.0.249.89 (38071) Safari 4.0.4 (531.21.10) Firefox 3.6 Internet Explorer 8.0.7600.16385 Flash Player WIN 10,1,51,95

IMPORTANT, THE ONLY RELEVANT COMPARISONS ARE BETWEEN PERFORMANCE OF THE SAME CONTENT ACROSS DIFFERENT BROWSERS AND OPERATING SYSTEMS. COMPARING FLASH CONTENT TO JAVASCRIPT CONTENT IN THIS TEST IS NOT VALID GIVEN THE DIFFERENCES IN THE CONTENT AND / OR FUNCTIONALITY.

Below are the pages / content run, and the amount of CPU they took to run. Lowest CPU usage is highlighted in green, and highest CPU usage is highlighted in red. I have also included a platform delta line, which shows the range in performance between the high and lowest CPU usage per operating system. Note, that as time passes, the content at some of the URLs may change, and not reflect the content tested.

Note: I am looking into why Internet Explorer is reporting 0% CPU usage on some of the JavaScript and Flash examples.

MacHeist Dynamic Canvas / JavaScript animation example

Mac Windows Google Chrome 95% 80% Safari 25% 80% Firefox 100% 40% Internet Explorer NA 0%

9elements JavaScript / Canvas Dynamic Animation Example

Mac Windows Google Chrome 100% 96% Safari 98% 104% Firefox 100% - 170% 72% Internet Explorer NA Did Not Work

Sublime HTML 5 HD Video Example

Mac Windows Google Chrome 85% 24% Safari 21% 104% Firefox 85% 80% Internet Explorer NA Did Not Work

HYPE Dynamic Flash Animation Example

Mac Windows Google Chrome 25% 16% Safari 17% 48% Firefox 23% 18% Internet Explorer NA 0%

Vimeo HD Flash Video Example

Mac Windows Google Chrome 88% 56% Safari 58% 64% Firefox 70% 64% Internet Explorer NA 104%

Overall Comparative Performance (shows how often each platform performed the best per content type)

Mac Windows HTML / JavaScript 3 6 Flash 2 4 All 5 10

I am not going to draw any definitive conclusions from this, but I do have some observations based on the results above.

• From these tests, Flash content does not perform consistently worse on Mac than on Windows.
• There is a wide range of CPU usage for HTML 5 video, depending on the browser / operating system it is being played back on, with Mac generally being slower.
• Canvas / JavaScript animations (at least those tested) seem to have high CPU usage, and generally run slower on Mac than on Windows (although not in all cases).
• Some of the HTML / JavaScript content would not run across all browsers.
• There seems to be some bug in the tested Flash Player when playing back video in Firefox. CPU usage would climb very high, and then drop. (I have reported this to the team).
• Availability of hardware acceleration for playing back video looks like it makes a huge difference in CPU usage (duh!).
• Regardless of technology (Flash and JavaScript / HTML), performance of rich media can vary widely depending on which browser / operating system it runs on.

AGAIN, AS I STATED BEFORE THE ONLY RELEVANT COMPARISONS ARE BETWEEN PERFORMANCE OF THE SAME CONTENT ACROSS DIFFERENT BROWSERS AND OPERATING SYSTEMS. COMPARING FLASH CONTENT TO JAVASCRIPT CONTENT IN THESE TESTS IS NOT VALID GIVEN THE DIFFERENCES IN THE CONTENT AND / OR FUNCTIONALITY.

Please keep comments on topic. Off topic comments will be moderated / deleted.

Update (March 10, 2010) : You can find similar (but separate) tests and discussions on this topic here and here.

Update (March 11, 2010) : Updated to make it clearer which Flash Player version was being used.