About Publishing Code Benchmarks

We often see code benchmarks. Some browser X HTML renderer is faster than browser Y renderer. Some JavaScript engine outperforms the competition by two folds.

While these benchmarks give a kind of instant gratification for the product, they always make me dubious coming from anyone. If the target is to outperform another browser, then I sense that nothing useful has really been accomplished. Even as a marketing technique, I don't think it's working.

When/if publishing a benchmark, focus on three things:

• How this new code outperform the previous versions of the code? It's good to show that we care about our product and that we want to be faster where/when it matters.
• How does this improve the user experience on some specific sites? Improving speed in controled environment like benchmarks is nice, but improving speed on real cases Web site is even better. Did it make the JavaScript-controled scrolling faster and smoother?
• How did we get there? What are the steps which have been taken to improve the code performance? The coding tricks and techniques used to make it faster.

These will be benchmarks blog posts I like to read. So as a summary

Good benchmarks show 1. Outperform your own code 2. Real websites improvement demos 3. Give Technical explanations.

Otsukare!