Yarn's Future – v2 and beyond

Hi everyone,

When the Yarn project started back in 2016, our landscape was very different from what it is now. Package locking was far from being a first class citizen in the Javascript ecosystem, and the time needed to run an install was ... well, it was what it was. Yarn's release shook the status quo and started a movement that ended up being beneficial to everyone, other package managers included. Now is the time to assess where we are, reinforce our strengths and patch our weaknesses.

This thread aims to expose our roadmap for the next major Yarn release, and let you know about significant changes that we plan to make regarding Yarn's design. The codename for these changes is Berry - that's how we'll refer to it during the next few months.

Roadmap

This section lists some of the work we have in progress. It's far from being a comprehensive list, but it gives a good idea of the kind of changes you can expect going forward:

Major changes

• The lockfile (and configuration) format will become a strict subset of YAML. In the process, the lockfile information will be slightly changed to account for some long-awaited changes (such as #5892 ).

• We'll add support for plugins, which will be able to alter various things - from adding new commands to hooking into the resolution / fetching / linking steps to add support for new package sources or install targets.

• Related to the plugin system, Yarn will become an API as much as a CLI. You can expect to be able to require it and start using its components in your script - no need to parse your package.json anymore, no need to run the resolution .. Yarn will abstract all those tedious tasks away.

• Support for both Node 4 and Node 6 will be dropped. We don't expect Berry to have stable releases until after Node 6 gets EOL (expected April 2019), by which point it won't matter anymore.

• The log system will be overhauled - one thing in particular we'll take from Typescript are diagnostic error codes . Each error, warning, and sometimes notice will be given a unique code that will be documented - with explanations to help you understand how to unblock yourself.

• Some features currently in the core (such as autoclean ) will be moved into contrib plugins. They'll still be supported, but might have a different release cycle than the standard bundle.

• The codebase will be ported from Flow to TypeScript. To understand the rational please continue reading, but a quick summary is that we hope this will help our community ramp up on Yarn, and will help you build awesome new features on top of it.

• The cache file format will switch from Tar to Zip, which offer better characteristics in terms of random access.

New features

• Nested workspaces (when one of your workspaces itself defines additional workspaces) will be supported out of the box. This will open up new exciting usages, especially when used together with git submodules.

• Running yarn run foo:bar (note the : ) will run the foo:bar script regardless of which workspace implements it. This will make working with monorepos a lot easier, as you won't have to constantly cd between directories anymore.

• Running yarn ./packages/my-package add foo will run yarn add foo into my-package . This should also contribute to make monorepos more pleasant, as you won't have to cd to add a new dependency somewhere.

• A new resolution protocol, workspaces: , allows you to force the package manager to link one of your packages against a workspace. This makes it impossible to accidentally use a version of a package obtained from the remote registry.

• A new command will appear, yarn constraints . This command will allow you to enforce constraints across workspaces - for example to require all your declared dependencies to use a unique version of a package.

• A new type of dependencies will be introduced, portals. Portals are like link: , but follow the chain of dependencies (whereas link: dependencies never have any dependencies, but don't require a package.json either).

• The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play :wink: ).

• Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

• Scripts will be able to put their arguments anywhere in the command-line (and repeat them if needed) using $@ . Similarly, scripts will have access to $1 , $2 , etc.

• The PnP hook will be able to read files directly from the cache Zip archives - not unlike the phar archives in php.

• Our cache will finally become fully atomic - multiple Yarn instances will be able to run concurrently on the same cache without risking corrupting the data.

And of course, a large amount of bugs are also expected to disappear as we modernize the codebase. A lot of those were caused by intricacies in the various components making up Yarn and, by cleanly separating them into standalone steps, we expect the overall logic to become much clearer and less error prone.

As you might guess, this isn't a comprehensive list - we plan Berry to be the big refactoring that Yarn had needed for a while now so many internal systems will be reworked. After that we expect following majors to become much smaller, as we'll have strong foundation for building new features and reworking the project one component at a time.

Yarn's philosophy

Now that we've discussed about most of the "What", let's discuss about the "Why". First, those are the values we always believed in and around which we continue to design most of our features.

Developing Javascript projects shouldn't leave the door open to surprises

This is our number one motto. We believe that Yarn should have a totally predictable and reproducible behavior. That your package manager has a responsibility to warn you when your project relies on unsafe patterns that could jeopardize your application at the worst possible time.

Developing Javascript projects should be easy

The complexity of the average Javascript project grown during the past years in order to accommodate with the ever increasing number of use cases people found. We believe that the package managers are in a unique position to help decrease some of the cognitive load and make it easy for their users to manage their projects and follow good practices.

Contributing to the Yarn ecosystem should be simple

Because a single project will never be able to satisfy all the needs from all developers counting on us, various projects spawned and grown, reinforcing Yarn rather than competing against it. Some examples are Create-react-app, Lerna, OpenCollective, or patch-package. We like that approach and want to give you the tools to do even more in a more efficient way. Not all features belong to the core, but being able to innovate and experiment is at the core of our philosophy, and we really mean it.

Goals for 2019

Now that our values are clear, what are the next steps we need to do in order to reach our goal? Note that some are long-term, some short-term, and that all are open to discussion (ideally in separate threads, otherwise it might get noisy ).

Yarn will become a development-first tool

We've been saying it for some time now, but now we're ready to start going for it. Package managers are not tools you should ever run on your production servers. The more code run there, the higher are the probabilities that something will eventually go wrong and bring your production systems down. Yarn being development-first means that we'll make it possible for you to reach the state where you clone a repository, and voilà. This includes putting the emphasis on Plug'n'Play , which has been designed with this specific use case in mind.

Yarn will be rewritten in TypeScript

Yarn is currently fully covered by Flow types, and to be honest it works pretty well for us - the key part being “us”. Because we want to make it as easy as possible for third-party contributors to shim in and help us maintain this awesome tool, we'll switch the codebase to TypeScript. We hope this will help making the codebase feel more familiar than the projects your already contribute to.

Yarn will become an API, and its internal components will be split into modular entities

This one is huge. At the moment, when working with Yarn, your only option is the command line interface. We don't offer primitives that allow you to take advantage of the complex logic we've implemented - whether it's the resolvers, linkers, or even accessing the configuration. This will change, and Yarn will be redesign to become an API first and a CLI second.

Yarn will be designed to support different install targets than Node

Package management is a problem that keeps on being reinvented - even Yarn itself did it in the past. We believe the reason for this is that all package managers need to layout the installed packages in slightly different ways in order for them to be read by the host. This unfortunately is rarely possible, and it ends up being easier to rewrite the package manager and discard the already-existing user experience and features. Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Overall compatibility will be preserved when possible

Such semver-major changes are by design backward-incompatible, but we'll make sure to keep them to an acceptable level. In particular, the core command ( yarn install , yarn add , yarn remove , yarn run ) will keep the same behavior, and your previous yarn.lock will be silently migrated. One important caveat: our installs will now use Plug'n'Play by default . We've proven during the past few months that the approach was sound, worked to trim all potential implementation issues, and finally discussed with project maintainers to figure out whether there was anything we could do to help them make sure everything was ready. Now is the time to take the plunge.

FAQ

When will it be released?

We'll open a new repository early February with the v2, the time to flesh out some remaining interfaces. Starting from then, we'll make Berry accessible simply by running yarn policies set-version berry within your project.

What will happen to the v1?

As mentioned before, Berry won't have a stable release until this summer, and work will continue on the v1 until then. Even after Berry comes live, the v1 branch will still stay well maintained for the foreseeable future, perhaps distributed through a codename. This will ensure that your applications that work now continue to work in the future, until you feel ready to move to the v2.

How can i help?

The biggest way you could help us at the moment would be to enable PnP ( yarn --pnp ) and report any issue to the maintainers of the packages that don't work yet (ideally with a PR if you have the time). Most of the work they would need to do is to make sure they use the standard require.resolve api, or offer a way to configure a different resolution strategy. At the moment the biggest blocker we have is the tsc default CLI, which doesn't do either (thankfully ts-loader is just fine - we actually use it in order to compile our bundles!).