August 23, 2019 | Soumya Swaroop Gupta, Nivedha Senthil

How Taiko compares to other browser automation tools

ThoughtWorks has been writing free and open source automated browser testing tools for over 15 years. Selenium RC was released in 2004, and WebDriver in 2007. Today, there are a lot of free and open source tools to automate browser based tests. Unfortunately, problems like flakiness and the cost of test maintenance have plagued this category of tests for a long time.

Over time, as users, we have opted for workarounds to deal with some of these issues. But sometimes all it needs is a different approach.

This year, the Gauge (by ThoughtWorks) team released Taiko 1.0, a simple NodeJS library to address some of the problems of browser-based automated tests.

In this post, we’ll compare Taiko with five popular open source browser automation tools

• Selenium
• Webdriver IO
• Testcafe
• Cypress
• Puppeteer

Since we are a part of the team that built Taiko, our intention to compare is to discuss our rationale behind each point of comparison (along with examples) and how we think Taiko’s approach helps in each instance.

While analysing, we considered these parameters for a holistic comparison

• Test maintenance
• Reliability (reduced flakiness)
• Performance
• Test frameworks integration
• Ease of test failure analysis
• Cross Browser support
• Language support

1. Test Maintenance

The goal: Tests should change only when there is a change in functionality. A change in the structure of a web page should not impact a test validating functional correctness.

However, most browser automation tools create tests that are hard to maintain. That’s because they use locators like XPath, ID's, CSS selectors to identify and perform actions on web page elements.

“Maintaining locators must be calculated as part of test maintenance cost.” - FireFox Test Engineering blog

These locators depend on the structure of a web page. The underlying structure can (and does) change frequently throughout the development process. Even minor changes in page structure break such tests. So, it increases the test maintenance costs.

Taiko’s approach

Taiko allows you to select elements on the page based on what you see. It's not based on the specific underlying structure that only a developer sees.

This means, to click that "Purchase" button you can use click( "Purchase"). It's based on the text on the screen and not based on the specific style, location or hidden identifier.

Multiple matches can be resolved with proximity selectors or a combination of them. Yet, if this is unsuitable, Taiko has fall back options. You can choose to select an element on the web page via XPath or CSS selector.

Benefits

Taiko’s API treats the browser like a black box. Tests written in Taiko are resilient to page structure changes as they minimise the need of using complicated locators.

Compare with examples

This example compares scripts automating the TODO MVC application. The tests currently pass for the React flavor of the TODO MVC. However, only tests written in Taiko pass when they are run against the AngularJS flavor of TODO MVC. This is because Taiko focuses on testing functionality and not the underlying page or framework.

Key Findings