Cross-Compiling in Scala

This feature comes up when talking about upgrading Scala versions. In this post, I will try to explain what it means and why should we use it.

Overview

The goal of this post is to help you update your application to the latest Scala version (2.12, 2.13, and counting), And by the end of this post you will know how to answer the following questions:

• Updating scala version — What’s the catch?
• How to cross-compile my libraries?
• How to update the Scala version in my application?

Updating Scala Version — What’s the Catch?

My first task in BigPanda was to upgrade our Scala stack from 2.11 to the (already quite old now) 2.12 version.

As I learned in this task, Scala is a version sensitive language, which means that code breaks between versions and one cannot use a 2.11 library in a 2.12 application.

Even more, I found that most of the artifacts we define in our build.sbt are defined using the %% syntax like this

groupID %% artifactID % revision

Which translates to:

groupID % artifactID_scalaVersion % revision

This means that our application requires that all it’s dependencies to be published in the same Scala version that it’s trying to use.

This process of updating to a new Scala version takes some time for libraries(see the famous spark 2.12 support here ) but the real problem lies in the in house dependencies which we have to update first.

How to Cross-Compile My Libraries?

My initial and naive approach was: Let’s upgrade our libraries to 2.12 and update all applications to 2.12 also, but it did not work.

The reason for the failure was in the numbers: In BigPanda , we’ve got about 10 Scala libraries and 10 complex applications that consume them. Migrating all of them at once is an insanely complex and risky operation that requires complex roll-back strategies and a large amount of time that we did not have.

The more practical way of doing this is by compiling our libraries to both 2.11, 2.12 using a feature called cross-compilation available in SBT.

It’s as simple as adding this line in your build.sbt:

crossScalaVersions := Seq(“2.11.11”, “2.12.7”)

And changing your build\publish command from:

sbt ++2.11.11 publish

to

sbt +publish

This tells SBT to publish it twice using the versions in crossScalaVersions . More info can be found here .

How to Update the Scala Version in my Application?

Now, this is what we really wanted to do. We choose the least critical service in our pipeline and decided to start there and iterate.

The process is as simple as writing this in your build.sbt file:

scalaVersion := "2.12.7"

And then making sure all regression tests pass (which is often not as simple).

After we deployed that service into production without any glitches, we felt much more confident about updating the rest of our services.

The Happy Ending

To be honest, it has been almost a year since we have done this task. we are currently at 9\10 applications migrated to 2.12 and we did not yet reach the part where we delete the cross-compilation into 2.11. It’s not an optimal ending, but a happy one in my perspective.

Conclusion

• Don’t be afraid to update your Scala version!
• Cross-compiling in SBT is a super cool feature that works out of the box and provides the ability to update your libraries to a new version while maintaining backward compatibility.
• Hope for the best, plan for the worst. Migrating all your services and libraries at once is a complex job with a lot of uncertainties.
• I recommend cross-compiling your libraries first to the required version and then migrating the applications one at a time in an iterative manner.