An update on rust/coreutils

An update on rust/coreutils

TLDR: we are making progress on the Rust implementation of the GNU coreutils.

Well, it is an understatement to say my previous blog post interested many people. Many articles, blog posts and some podcasts talked about it! As we pushed coreutils 0.0.12 a few days ago and getting closer to the 10 000 stars on github, it is now time to give an update!

This has brought a lot of new contributors to this project. Instead of 30 to 60 patches per month, we jumped to 400 to 472 patches every month. Similarly, we saw an increase in the number of contributors (20 to 50 per month from 3 to 8). Two new maintainers (Michael Debertol & Terts Diepraam) stepped in and have been doing a much better job than myself as reviewers now! As a silly metric, according to github, we had 5 561 clones of the repository over the last 2 weeks!

The new contributors focused on:

• Performances. Now, some binaries are significantly faster than GNU (ex: head, cut, etc)
• Adding missing binaries or options (see below)
• Improve the testsuite: we grew the overall code coverage from 55% to 75% (in general, we consider that a 80% code coverage on a project is excellent).

• Refactoring the code to simplify the maintenance. Examples:

  • Using the same code for permissions for chgrp and chown
  • Managing error the same way in the various binaries - (Kudos to Jeffrey Finkelstein for the huge work)
  • Improving the GNU compatibility (thanks to Jan Verbeek, Jan Scheer, kimono-koans and many others)
  • Move to clap 3. Upgrade by Terts which unblocks us on various problems.

Closing the gap with GNU

As far as I know, we are only missing stty (change and print terminal line settings) as a program.

Thanks to some heroes, basenc, pr, chcon and runcon have been implemented. For example, for the two last programs, Koutheir Attouchi wrote new crates to manage SELinux properly. This crate has been used for some other utilities like cp, ls or id.

Leveraging the GNU testsuite to test this implementation

Because the GNU testsuite is excellent, we now have a proper CI using it to run the tests. It is pretty long on the Github action CI (almost two hours to run it) but it is an amazing improvement to the way we work. It was a joint work from a bunch of folks (James Robson, Roy Ivy III, etc). To achieve this, we also made it easier to run the GNU testsuite locally with the Rust implementation but also to ignore some tests or adjust some error messages (see build-gnu.sh and run-gnu-test.sh).

Following a suggestion of Brian G, a colleague at Mozilla (he did the same for some Firefox major change), we are now collecting the history of fail/pass/error into a separate repository and generating a daily graph showing the evolution of regression.  At this date, we have, with GNU/Coreutils 9.0:

Total 611 tests

Pass 214

Skip 84

Fail 298

Error 15

We are now automatically identifying new passing tests and regressions in the CI.

For example:

Warning: Congrats! The gnu test tests/chmod/c-option is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/silent is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/umask-x is now passing!
<br />Error: GNU test failed: tests/du/long-from-unreadable. tests/du/long-from-unreadable is passing on 'master'. Maybe you have to rebase?
[...]
<br />Warning: Changes from master: PASS +4 / FAIL +0 / ERROR -4 / SKIP +0


This is also beneficial to GNU as, by implementing some options, Michael Debertol noticed some incorrect behaviors (with sort and cat) or an uninitialized variable (with chmod).

Documentations

Every day, we are generating the user documentation and of the internal coreutils.

User documentation: https://uutils.github.io/coreutils-docs/user/ Example: ls or cp

The internal documentation can be seen on:https://uutils.github.io/coreutils-docs/dev/uucore/
For example, the backup style is documented here: https://uutils.github.io/coreutils-docs/dev/uucore/backup_control/index.html

More?

Besides my work on Debian/Ubuntu, I have also noticed that more and more operating systems are starting to look at this:

• Brew is proposing coreutils (with the latest version) https://github.com/Homebrew/homebrew-core/pull/93404
• Sandro is working to be able to run NixOS on Rust/Coreutils. See https://github.com/NixOS/nixpkgs/pull/116274 for the interesting discussions there.
• Apertis (Linux distro for the automotive industry) has forked my Debian patches to make it the default coreutils https://gitlab.apertis.org/pkg/rust-coreutils/-/tree/apertis/v2023dev1/debian
• I also noticed an interest from Redox OS.

In parallel, https://github.com/uutils/findutils/, a rust dropped-in replacement for find, is getting more attention lately! Here, the graph showing the evolution of the program using the BFS testsuite (much better than GNU's).

What is next?

1. stty needs to be implemented
2. Improve the GNU compatibility on key programs and reduce the gap
3. Investigate how to reduce the size of the binaries
4. Allow Debian and Ubuntu to switch by default without tricky manipulation

How to help?

I have been maintaining a list of good first bugs for new comers in the repo!

Don't hesitate to contribute, it is much easier than it seems and a terrific way to learn Rust!