Many enthusiasts want a fast browser, and they want to consistently and easily measure the performance of different browsers. There are lots of “benchmarks” in the history of software, and lots of benchmarks around browsers. The challenge with benchmarks is generally understanding what the benchmark measures and deciding whether those measurements are important to customers, developers, and you.
This blog post takes a closer look at some of today’s most common benchmarks. We’ll provide context on their origins, talk about what they test, and show you what browser subsystems each benchmark executes.
Celtic KaneCeltic Kane is generally credited with being the first JavaScript benchmark. It was originally created in 2006 by a web developer (Sean Patrick Kane) and became popularized after the benchmark was featured on Digg.com. Celtic Kane consists of eight simple test cases which are each run once and measured in milliseconds. Celtic Kane claims to focus on core JavaScript and DOM performance; however the DOM tests inside Celtic Kane don’t actually measure the DOM.
WebKit SunSpider
WebKit SunSpider is one of the most commonly cited benchmarks and has become a focus for the industry. Given that SunSpider focuses on JavaScript performance and not real world scenarios that matter to customers this is unfortunate. SunSpider is developed by the Apple WebKit team with the goal of measuring real world JavaScript performance. There are 26 test cases which attempt to measure the performance of individual JavaScript features including arrays, math, and regular expressions. The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others.
Google V8
Google’s V8 was designed to help Google tune the V8 JavaScript engine. The benchmark carefully targets the JavaScript subsystem and measures some computationally complex scenarios such as OS thread scheduling, cryptography, and ray tracing in JavaScript. Though well designed, some of the V8 benchmark tests were programmatically generated from Scheme language benchmarks and do not use programming language patterns commonly used by JavaScript developers. While academically interesting these scenarios are not representative of how the JavaScript language is used or fairly represent patterns across the web.
Mozilla Dromaeo
Mozilla’s Dromaeo is one of the best engineered JavaScript and DOM benchmarks available today. It was originally written by John Resig and the Mozilla Foundation to support Firefox development and is a very effective tool for browser vendors who need to measure the performance of individual API’s. In addition to pure JavaScript and DOM tests, Dromaeo measures a few additional areas including common jQuery and Prototype features along with CSS Selectors. Dromaeo also provides a test harness for the WebKit SunSpider and Google V8 benchmarks.
Flying Images
The Flying Images demo released with the first Internet Explorer Platform Preview has unexpectedly become a “hardware accelerated graphics” benchmark across the web. Our intent with Flying Images was to show how the GPU can impact the HTML4 websites of today, not create a benchmark. That said it does fairly represent the underlying coding patterns found in today’s common AJAX applications. Flying Images uses standard HTML4, JavaScript, and CSS to programmatically move images around on the screen every 1/60 of a second (considered real-time on a 60hz display). It manipulates the DOM, changes CSS properties, and causes the entire page to go through the layout process.
FishIE Tank
The FishIE Tank demo released with the third Internet Explorer Platform Preview was treated by some bloggers and technology reviewers as a benchmark because of the visually compelling scenario. FishIE Tank merges both traditional HTML coding patterns with HTML5 Canvas. It uses HTML5 Canvas to animate fish around the screen with a JavaScript based collision detection system, and HTML elements and common HTML coding patterns such as setting innerHTML to display an FPS meter over the tank. This demo is representative of the coding patterns that we’ll see as developers build applications that merge the established HTML and emerging HTML5 constructs.
Psychedelic Browsing
The Psychedelic Browsing demo has received over 500,000 page views over the last five weeks since the fourth Internet Explorer Platform Preview was released in August. Psychedelic Browsing spins a color wheel as quickly as possible on the screen and was designed to stress a common HTML5 coding pattern that uses JavaScript and the DOM to draw directly into an HTML5 Canvas while playing an HTML5 Audio element in the background. We have seen this pattern start to quickly emerge over the last six months as Canvas has seen broad adoption. This is representative of where many web applications are headed.
We believe that to build a fast real world browser you have to start by understanding the real world patterns and design the right systems around those patterns. You can’t build a fast real world browser by optimizing for benchmarks. And you can’t accurately assess a browsers performance by looking solely at the results from benchmarks. When used for engineering purposes, benchmarks can be an effective tool to help understand the implications of code changes. That’s where the value ends though.
As an industry we have to care be careful that we don’t lose sight of what’s important – the real world customer scenarios of today and the HTML5 scenarios of tomorrow.
Jason Weber
Lead Program Manager, Internet Explorer Performance
More...