At Cookpad Global, we compliment our unit-tests with end-to-end UI-tests. For the iOS automation solution, we currently utilise the XCUITest framework.

In this post I will share about some of the problems we faced, and how we at Cookpad have been overcoming them to achieve a readable, reliable and maintainable end-to-end UI-test solution using XCUITest.

Flaky tests

Unreliable tests, also known as a ‘flaky tests’ are typically characterised by a test that can both pass and fail on subsequent executions without any changes to code. Such tests can introduce wastage in the software delivery life-cycle when verifying false positives, maintenance, and re-execution becomes required.

Network

Perhaps one of the most notable drawbacks of creating an automated end-to-end testing solution is the reliance on external dependencies, in our case for example, a live server environment. An extra flaky factor can be introduced when a test attempts to assert against or interact with UI that is reliant on server content which may arrive late, or not arrive at all.

Animation and interaction delays

Other flaky factors such as animations and interaction delays require consideration too. It’s not an uncommon sight when investigating failed tests to see that it was caused by an interaction or assertion that occurred when the app was not quite ready, for e.g. tapping while the element exists, but before it was hittable enough to register the tap.

These are just a couple examples of common flaky factors you may encounter when UI-Testing, but they aren’t the only factors. Things such as general assertion timings, test execution order and even bugs in the test framework can also introduce flakiness. However, a shared concept for a range of timing based flaky factors exists that can be utilised to better improve the reliability of tests, waiting.

A brief history of waiting in XCUITest

There are a few approaches available to wait in XCUITest, I will briefly talk about some of the more common approaches I have seen utilised.

XCUIElement.waitForExistence(timeout:) will wait explicitly to a given timeout, and return a boolean per the element’s existence property.

if app.buttons["identifier"].waitForExistence(timeout: 5) {
    // do some stuff 
}

XCTestCase.expectation(for:evaluatedWith:handler:) and XCTestCase.waitForExpectations(timeout:handler:) can be used to wait explicitly for expectations using predicates for example, allowing for more flexibility, and even an optional completion handler to be invoked on success or timeout.

expectation(
    for: NSPredicate(format: "exists == true"),
    evaluatedWith: app.buttons["identifier"],
    handler: .none
)

waitForExpectations(timeout: 5)

Xcode 8.3 introduced the XCTWaiterclass, which expanded on the former and allows us to better handle wait results, and provided the ability to wait for multiple expectations.

let expectation = expectation(
    for: NSPredicate(format: "exists == true"),
    evaluatedWith: app.buttons["identifier"],
    handler: .none
)

let result = XCTWaiter.wait(for: [expectation], timeout: 5.0)

XCTAssertEqual(result, .completed)

Additionally, and perhaps the least optimal solution in most cases, sleeping Thread.sleep(forTimeInterval:).

sleep(10)

Waiting at Cookpad

At Cookpad, we wanted an extension method on XCUIElement similar to XCTestCase.waitForExpectations(timeout:handler:) to make tests readible, but we also have more expectations to wait for than just existence, and we didn’t want to create multiple methods to do very similar things e.g. waitUntilHittable, waitUntilLabelMatches etc.

Additionally, we didn’t want to sleep as an expectation might occur before the timeout and we waited too long, or the opposite, and we didnt wait long enough and spent time verifying false positives. As a result, we created a solution utilising take-aways from all of the aforementioned techniques.

The base

First we created a base XCUIElement extension method called wait, which takes an expression block returning a boolean, with an XCUIElement receiver, and we wait for the expression block to be true using XCTWaiter.

This allows us to write statements such as:

app.buttons["identifier"].wait(until: { $0.exists })
app.buttons["identifier"].wait(until: { $0.label == "button_text" }
app.buttons["identifier"].wait(until: { button in
    button.exists && button.label == "button_text"
})

This is much more flexible, but not as readable as we would like.

The magic of keypaths

Now we have our base wait method, we can create a wrapper with a much cleaner declaration by forming the expression using an XCUIElement keyPath, and an equatable match value.

The keyPath in our case allows us to reference an instance property without neccessarily knowing which one, and using an equatable value allows us to conveniently create expectations against said property.

app.buttons["identifier"].wait(until: \.exists, matches: false)

Now this is looking much better in regards to readibility and usability, however, there is still room for improvement!

The final touches

Implement a new wait method which calls the previous declaration, where matches is always true.

This wrapper further improves the readability of our test cases for a use-case we find most common, waiting for an XCUIElement property to be true.

app.buttons["identifier"].wait(until: \.isSelected)

Usage

This approach can be utilised to explicitly wait for a dynamic range of expectations. The following examples are a handful of the more commonly used methods from our UI-test solution.

isHittable

app.alerts.buttons.element(boundBy: 1)
     .wait(until: \.exists)
     .wait(until: \.isHittable)
     .tap()

isEnabled

app.button["identifier"].wait(until: \.isEnabled)

isSelected

app.button["identifier"].tap()
app.button["identifier"].wait(until: \.isSelected)

Custom properties

app.button["identifier"].wait(until: \.isDisplayed)

Miscellaneous

app.staticTexts["identifier"].wait(until: \.label, matches: input)
app.tables["identifier"].wait(until: \.cells.count, matches: 0)

The key takeaway from the above examples that I am aiming to demonstrate is the variety. We believe this versitile approach allows us to cleanly and explicitly wait for a wide range of expectations.

Preferences

The solution we’ve come up with works great for our specific needs and allowed us to cleanly improve the reliability of our tests, but out of the box this may not be the be-all and end-all solution for everyone.

For example, we return self for the ability to chain methods, but perhaps this isn’t everyone’s cup of tea.

app.buttons["identifier"]
    .wait(until: \.isEnabled)
    .wait(until: \.label, matches: "button_text")
    .tap()

It could alternatively return nothing at all, or perhaps return a boolean per the wait result and have a behaviour more akin to XCUIElement.waitForExistence(timeout:), for example:

if app.buttons.element.wait(until: \.exists) {
    // do some stuff 
}

Furthermore, we haven’t concerned ourselves yet with handling the result of the wait beyond success or failure, and we haven’t implemented any completion handlers either, purely for lack of need at the moment. However, these are all within the realms of possibility per the needs of your app/tests.

Conclusion

We love the readability, flexibility and reliability that such methods provide. This post is just one part of how we achieved readable, reliable (flaky-free) and maintainable end-to-end ui-tests for the iOS platform at Cookpad Global, and we look forward to sharing more about our work in future posts.

Complete extension (with documentation) available here