Choosing the Right Continuous Integration Tool

A typical software development process is based on an agile framework with a need to roll out quality products at a fast pace. Software delivery tools are also evolving to keep pace with challenges like frequent changing requirements, evolving design, and continuous integration with test-driven development. Continuous integration (CI) and continuous delivery (CD) are not only the latest buzzwords but also a highly effective and practical way of developing and delivering quality products. Today, we are flooded with many CI tools and choosing the right tool is becoming difficult. Many organizations take the easy route by picking a popular tool, but this strategy may be risky since the chosen tool may not meet their needs in the long term.

This article will help you choose the right CI tool as per your business needs. Let us first look into the popular CI tools on the market. The below table gives a list of CI tools (not ranked in any order) that are popular today.

Jenkinsis an open-source automation server written in Java. It helps to automate the non-human part of the software development process, with continuous integration and facilitates the technical aspects of continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat

TeamCityis a Java-based build management and continuous integration server from JetBrains Inc. It is a commercial software and licensed under a proprietary license.

Travis CIis a hosted, distributed continuous integration service used to build and test software projects hosted at GitHub. Open-source projects can be tested at no charge via travis-ci.org. Private projects can be tested at travis-ci.com on a fee basis.

GoCDis an open-source CI server, which is used in software development to help teams, and organizations automate the continuous delivery of software. It supports automation of entire build-test-release process from code check-in to deployment.

Bamboois a CI server that can be used to automate the release management for a software application by creating a continuous delivery pipeline. It is developed as Atlassian Inc. with propriety license for the private projects but a free license for open-source projects.

GitLabis a web-based Git-repository manager with wiki, issue tracking, and CI/CD pipeline features, using an open-source license, developed by GitLab Inc. It can be used freely for limited used for basic features but license fees are required for advanced features.

CircleCI's continuous integration and delivery platform makes it easy for teams of all sizes to rapidly build and release quality software at scale in the cloud or behind the firewall. Excellent support for Linux based software development. Supports all open-source version control systems

Codeship is a fast and secure hosted Continuous Integration service that scales with business needs. It supports GitHub, Bitbucket, and Gitlab projects.

Werckerempowers organizations and their development teams to achieve continuous integration and continuous delivery (CI/CD) goals with micro-services and Docker. This is achieved via Wercker’s container-centric and cloud-native automation platform comprised of their local command line interface (CLI), online SaaS platform and API. Wercker is hosted on Oracle Public Cloud for creating containerized pipelines.

Jenkins Xhas the concepts of Apps and Environments built in. Jenkins X does not aim to replace Jenkins since Jenkins is embedded as a pipeline engine as part of the installation. It is a one of the fast emerging CI/CD solutions for modern cloud applications on Docker and Kubernetes.

Shippableis lightweight CI, CD, and DevOps automation platform with Docker support that simplifies provisioning, building, testing, and deploying any application anywhere. It comes with a propriety commercial license with support for Github and BitBucket repositories.

NeverCodeis a cloud-based CI and CD server that automates the process of building, testing and distributing mobile applications. Claims to help developers develop apps 20% faster with the leading CI/CD solutions.

Essential Technical Features

A CI tool must have features which cater to both on-premises and cloud development, as follows:

Security Enablement

Role-based access control, SSO based authentication, CSRF Protection, Slave/Node/External Runners to Master Access Control, protection from malicious builds and users

Container Support

Support to build and pull Docker images from various Docker repositories (both private and public), Support for Kubernetes cloud, running builds on Kubernetes nodes, support of persistent storage of logs and workspace, deployment of built Docker images to Kubernetes managed cloud

Version Control Software Support

Support for various version control system and repositories like Subversion, Perforce, Git, Rational Clear Case, Razor, SourceAnywhere etc. hosted on-premises or in the cloud.

Easy Complex Pipleines

Ease to create complex pipelines. There could be requirements which could result in a complex release steps and a CI tool should meet these requirements. Such requirements might mandate pipeline to have multiple forks and joins based on conditions and trigger remote pipelines and workflows.

Good User Interface

The user interface should be web browser based to check the build consoles, build configuration, pipeline as a code in YAML format, build time history, build marking with various badges, support for various views like pipeline view, job view, workflow view etc,

Instant report views for various builds, build comparisons, ease of global configuration, ease of third-party tool configuration, etc.

Notifications Support

Support for notifications on various integration stages via Email, Hipchat, Slack, Stride, Flowdock, Campfire etc.

Support for Build Automation Tool

Different projects have different type of build tools requirement. A CI tool must be flexible enough to support all types of build tools like Make, Shell Scripts, Ant, Maven, Gradle, Rake, Grunt, Gulp etc.

Artifacts Management Support

Feature to store, retrieve the build information, build/test artifacts pull/push from/to Apache Nexus, JFrog Artifactory or any other artifacts repositories via plugins, steps or REST calls.

Multiple Programing Language Support

Support for various programming languages like Node.js, Java, Ruby, Python, Golang, Groovy, Shell Scripts etc.

Integrations with Public Clouds

It should be able to do pipeline deployments to various Oracle Cloud – Classic, Oracle Cloud Infrastructure, Google Compute Engine, and AWS with ease.

Monitoring Support

Monitoring of various build environment parameters like slave’s memory, free disk space, builds, number of executor threads, number of nodes, time sync with master, etc.

Easy Integration with Third Party Software

Automated filing and updating of bugs to JIRA, incident filing in Pager Duty and BugZilla. Integration of static analysis tools, legal compliance tools for open-source and for the in-house built artifacts.

Library of Re-usable Code

Public store of various plugins, usable build steps, which could be open-source or commercially available.

Business Requirements

Apart from the technical features, it is also essential to compare every CI tool with respect to the business requirements. Below are few such points that could be useful for such evaluations.

Open-Source vs. Commercial License

The main driver of decision here is the budget. Choose one of the popular CI tool if budget is available for license fees and there is a need for online/offline support for pipeline maintenance. Teamcity, CircleCI, Bamboo, GitLab-EE, Wercker and Codeship are few of the popular CI tools in this category. Some of these tools charge fees based on the number of users, remote agents/nodes/runners and number of features while some have fixed license fees. Utmost care should be taken to not be stuck with a CI tool that does not have a good plan to adopt to emerging development technologies.

Alternatively, it makes sense to go for CI tools, which are open-source like Jenkins. These are mostly community driven projects with lot of plugins and support available via online tutorials, blogs, chats and forums. There are organizations which provide commercial support for open-source CI tool, for example, CloudBees Inc. provide commercial support for the Jenkins.

On Premises vs. Cloud Hosted

Cloud software architecture is quite different from an on-premises traditional software development. Cloud based solutions are largely based on micro services architecture with software pieces strewn across the network. They are implemented using container tools like Docker and deployed to Kubernetes via offerings from Google, Amazon or Oracle etc. Tools like Gitlab and Wercker are perfect for Docker based micro services with GKE or custom Kubernetes support where as tools like Jenkins can be enhanced via the plugins to cater for these use cases.

CI servers like Jenkins requires an on-premises server hosting with slaves/nodes (virtual machines) attached to it. This requires efforts to install, monitor and maintain the system.  However, a cloud-hosted solution is mostly ready to use with minimal configuration efforts, offers infrastructure reliability and can be scaled up/down based on the job load. Long term operation costs of an on-premises hosting could be less expensive than the cloud hosted solution. Other deciding criteria could be legal and statutory requirements where the artifacts or data should not be posted to cloud but should be accessible to the compliance authorities locally. For smaller setups or startups, the cloud based hosted CI solution is better where basic essential features are available for small set of users with minimal or no fees.

Choose Multiple CI Tools

Can one size really fit all? Not really. A popular CI tool may meet 80% of the automation requirements but not fit the other 20%. Jenkins, for example, can meet almost all the requirements but it requires manual effort to monitor the Jenkins server, its slave, understand inter dependencies amongst plugins and upgrade various plugins from time to time. There are CI tools like Wrecker which works perfectly if a product is cloud based but may require support for an on-premises software product.

An organization could have multiple projects of different kinds, which could range from a monolithic big software system to a micro service based architecture and one CI tool may not be sufficient for all the scenarios. So, it is better to choose multiple CI tools for various needs instead of fitting all into one CI tool. Such a strategy is useful for the business continuity too where projects will not be impacted if a support from a CI tool is not good or the CI tool is discontinued.

Migration From a CI Server

A CI pipeline could be defined “as a code” that directs the CI server to use a specific build environment, build steps, easy integration with third party applications etc. Most of the pipeline as a code uses YAML format, which could be compatible to other CI tools too. However, custom inbuilt YAML step might not be portable to other CI tools, so, care should be taken to implement the build and deployment steps as generic as possible so that if the need arise, the entire pipeline could be shifted to another CI server.

Conclusion

Yes, there are a lot of CI/CD tools available today and it is difficult to choose a best amongst many considering the fact that many tools do not provide all the required features. Therefore, it is suggested to choose a CI solution with the help of the pointers given in this article with a condition that it should not only meet the current integration requirements but should also evolve along with your product’s future roadmap.