Automatic quality assurance measures in iOS/tvOS projects with Fastlane, Danger...

Are you working in a development team on an extensive and dynamic iOS project? Then Fastlane, Danger, SwiftLint and GitLab can help you to ensure quality and save time.

Introduction

Waipu.tv is an IPTV video streaming platform that has become the market leader within two years. Its vision is to continuously improve the product through constant innovation and the incorporation of customer feedback. We had the pleasure to support waipu.tv with the development of the iOS / tvOS apps and it was a lot of fun.

The implementation of the video streaming app waipu.tv for iOS (iPhones) began in autumn 2015. Over time, iPadOS (Universal App) and later tvOS (Apple TV) were added to the supported platforms.

Since the beginning of the development (then Swift 1, iOS 7 / iOS 8 –  today Swift 5 – iOS 11- iOS 14) the code base has been growing and changing constantly.

This is caused by:

• New features
• Bug fixes
• Refactoring of existing code
• Updates and maintenance of connected services
• New major versions of iOS, iPadOS, tvOS
• Updates to the version of the Swift programming language
• Integration and updates of 3rd party SDKs

We develop in two-week sprints at the end of which a new version of the apps is created. In order to ensure the quality of the rapidly growing code base and to keep the number of bugs low, the development team has introduced various automated measures beyond standard Unit- and UI-Tests, which are described below.

Since every code change finds its way to the main-branch of the git repository via a merge request, we have implemented various mechanisms at this point in order to carry out basic checks automatically.

Automatic checks in the merge request

We use GitLab on premise as version control system. It is entirely based on Git and, among many other things, offers options for code reviews. New features and bug fixes are developed on feature branches. As soon as a new feature has been developed, it is made available as a merge request for review by at least one developer.

When creating a merge request, developers should be aware that a careful review entails a time investment for the reviewer and the code should meet team standards.

The following part explains how compliance with the team conventions can be checked using automatic checks and consequently a basic quality of the merge request can be ensured. This saves the team unnecessary discussions (and time) in merge requests, e.g. about formatting or missing changelog entries, as the rules are defined before and automatically checked. 

Continuous Integration with Fastlane and Gitlab CI

We use Fastlane for build, test, release and code signing.  The configuration takes place in the Fastfile, in which so-called “Lanes” are defined.

A “Release-Lane“ could, for example, consist of the following sub-steps:

• Run UnitTests and UITests
• Make a release build of the app
• Sign the app
• Upload the app to App Store Connect

A detailed description of continuous integration with fastlane and Gitlab-CI can be found in this blog article by Sebastian.

As soon as a new merge request is created GitLab triggers a pipeline that compiles the merge request code via Fastlane and executes unit tests. The build itself takes place on a separate macOS build server on which a GitLab Runner is active. It receives and executes the build jobs. If an error occurs when compiling or executing the tests of a merge request, it cannot be merged. This ensures that only compilable, unit-tested code can be merged.

Automatic checks for the merge request are also carried out at this point. The tools used for this purpose are presented below.

SwiftLint

On the project website SwiftLint is defined as follows:

A tool to enforce Swift style and conventions, loosely based on GitHub’s Swift Style Guide.

SwiftLint hooks into Clang and SourceKit to use the AST representation of your source files for more accurate results.

SwiftLint is a widely used open source tool with a large code style rule set that checks whether these have been adhered to in the written code. There are numerous additional opt-in rules that can be activated. There is also the possibility to create your own rules (customRules) or to participate in the project itself.

The rules are configured in a .swiftlint.yml file:

If SwiftLint is introduced to an existing project, the number of errors/warnings can be very high. It makes sense to gradually make parts of the code SwiftLint-conform by configuring included/excluded directories.
Another help can be swiftlint autocorrect (swiftlint --fix since version 0.43), which automatically fixes easy-to-solve warnings and errors. Typically, the .swiftlint.yml file develops in the course of the development and growing codebase of the project and “custom rules” are introduced. To see SwiftLint warnings and errors in Xcode, it can be integrated into Xcode in the build phase settings.

Danger

Danger is a software originally written in Ruby (implementations are also available in JS, Swift, Kotlin and Python) that runs during the CI process (or standalone). Danger can carry out pre-made or self-developed (custom) code checks for automatic reviews of merge requests that are stored in the Dangerfile and leave comments in the merge request. Typically, this Dangerfile develops because the team keeps finding new standards and ideas for automatic checks over time, e.g. to avoid a problem that has occurred before (we often added new checks as a measure from the scrum retrospective meeting). In this way, the team norms can be stored in the Dangerfile and the reviewer can concentrate on the “higher” problems when reviewing the merge request.

A comprehensive list of ready-to-use danger plugins (also for many other areas of application such as Android apps) is available on GitHub.

Examples of checks in the Dangerfile:

• danger-ruby-swiftlint: SwiftLint plugin for Danger. Checks the Swift style.

• danger-todoist: Warns if a TODO is left in the merge request (this potentially indicates that the developer has forgotten something). Our team has decided not to check in any TODOs.

• danger-ios_logs: Warns of NSLog and print statements in the release code. 

• danger-xcode_summary: A Danger plugin that shows all build errors, warnings and unit tests results generated from xcodebuild.
• danger-xcprofiler: If compilation times of each methods are exceeding the thresholds, Danger adds an inline comment to your merge request.
• Danger-Slather: A Danger plugin that shows the code coverage of a Xcode project and file by file using Slather. It can add warnings or fail the build if a minimum coverage is not achieved.

• Custom-Check for iPhone and iPad branches in the code: If there is a switch in the code that contains various implementations for iPad and iPhone, a message is displayed that the changes should be tested on the iPad and iPhone.

• Custom-Check for changelog modification: Checks whether the changelog file has been modified.  ​​Usually a merge request contains a feature or a bug fix for which there should also be a corresponding changelog entry. This way you can ensure a well-maintained changelog for every release.

• Custom-Check for info.plist modifications: If the info.plist has been edited (e.g. the version number has changed) a message with the hint that further info.plist files may have to be adapted is displayed (e.g. for NotificationExtensions or the build scheme of the tvOS app part of the project).
• Custom-Check for changes in XIB-files (Xcode Interface Builder files): If a XIB-file has been edited, a message that the developer should add a screenshot of the modified UI to the merge requests displayed. Also prevents accidentally edited XIB files from getting into the productions code.

Heres an example of a Dangerfile:

Configuration: Bringing it all together

The configuration for performing automatic checks for a merge request described below assumes that the project has been automated with Fastlane. Fastlane is implemented in Ruby. We will use Bundler to run FastLane. Bundler is a tool that helps to provide a consistent environment for Ruby projects. Bundler reads the so-called Gemfile in which the dependencies on Ruby gems are defined. This means that a consistent environment can be provided at any time on any system (e.g. on the build server or on the development computer) with the required gems in the appropriate versions.

Heres an example of a Gemfile:

Fastlane comes with its own action for Danger. This action is the entry point and it is executed after creating a merge request in GitLab in the „test“ stage (bundle exec fastlane run danger).  To do this, the following configuration must be added in the .gitlab-ci.yml:

After GitLab has called Danger, it reads its configuration from the Dangerfile and carries out the appropriate checks. SwiftLint is also configured as a plug-in in the Dangerfile. This reads its configuration from the .swiftlint.yml file.

In action: An example merge request

The merge request on the following screenshot shows Danger and GitLab in action. Danger (and the SwiftLint-Plugin) found a problem with the number of parameters in a function. The other checks were carried out successfully:

• The changelog file was adjusted ✔️
• Ticket number is included in the title of the merge request ✔️
• No TODO/FIXME were found in the code ✔️

A merge request in GitLab that was validated by Danger

What are the advantages?

• Direct automated feedback after creating a merge request. New team members automatically learn the norms already set by the team.
• Code conventions are checked automatically via “code”.
• Useful automated hints for the reviewer (e.g. „do an extra check on the iPad“)
• No unnecessary friction in the team due to “indentations“, „code style“, … – the machine decides – the team defines the rules beforehand in the code.
• If the team recognizes a certain error pattern, e.g. in the scrum retrospective, a corresponding check can be implemented as a measure. In this way it can be avoided that the same error occurs again and again.
• The reviewer of the merge request has more time to focus on the core of the merge request.

Multiple Devices Test

As a mobile developer, we usually test our code on a current device or simulator with the latest operating system version. As a rule, however, older devices with older operating system versions are also supported. In our project there is also the fact that parts of the code are shared with the app for the tvOS platform. This can lead to bugs or crashes if, for example, switches for the OS version are implemented incorrectly. Problems can also occur in used 3rd-party libraries that are regularly updated (and potentially contain bugs with every update).

Manual testing with a defined set of device-/OS-combinations takes a lot of time. For this reason, we have chosen the following way to automatically check the functionality with a defined list of devices – we call it the Multiple Devices Test.

GitLab offers so-called schedules (comparable to a cron job) with which tasks can be carried out periodically. We have created a schedule that carries out all Unit- and UI-Tests of the project every night on a defined list of device-/OS-combinations.

Since this test run takes more than three hours, it would be too time-consuming to execute it within each merge request. The test is carried out via Fastlane. We use the Fastlane action Scan which offers the possibility of executing tests on a list of devices. First we define the list of test devices with which the test should run in the Fastfile:

Then we create a new lane test_nightly_multiple_devices in the Fastfile:

Now we add the section test_nightly_multiple_devices to the .gitlab-ci.yml.  This allows the multiple devices test to be started via the GitLab schedule.

Now the corresponding schedule can then be created in Gitlab:

GitLab Schedules Overview

All tests are now regularly carried out on the defined device-/OS-combinations and, in the event of an error, a message is posted via e-mail and in the slack team channel.

Advantages of the Multiple Devices Test

• Automatic check that the code does not crash on the defined platforms – no more: “Oops, the app crashes on iOS 11 immediately after starting”
• Problems with the UI on older OS versions can be recognized by the fact that the UI-Tests are also run on these OS versions
• Any problems with 3rd-party libraries on older operating system versions can be discovered
• Shared iOS/tvOS code is being tested on both platforms

The multiple device test helped us to identify bugs and problems before the release. The implementation effort is low and gives the team an additional insurance to deliver the best possible release.

That’s it ?

Thanks for reading! Maybe the measures described will find their way into the development process of your project.

Are you looking for support in your app development process? Have a look at our offerings!