Github rfc: make cargo install extensible by ashleygwilliams · Pull Request #237...

ashleygwilliams commented on Mar 27, 2018

for context i would really like this (if considered to be viable) to be blocked on #2196 because i believe the pattern that this RFC introduces builds the affordances in a way that mitigates much of what i believe could be seen as a downside of this RFC

est31 commented on Mar 27, 2018

Currently, cargo install tracks what files it installs, and supports cargo uninstall; this extension mechanism does not hook into that. Potentially, install.rs could emit a list of installed files and let cargo install install them, producing a log of those files for later uninstallation. However, we do not intend for cargo install to become a full-featured package management mechanism; rather, we expect cargo install to work analogously to make install. While the occasional package provides a make uninstall, few developers expect such a mechanism.

I guess I'm one of those who think that clean uninstallation is an important feature. I hope the statement that we are few is wrong.

I avoid make install like the plague and if I actually have to use it, if just invoking some binary in some build dir is not enough, I create a separate root fs and let it install inside there. This is a major hassle that I do just to have the simple pleasure of not getting my file system messed up with random stuff where I don't even know where it came from.

Currently, Cargo solves this in a beautiful way: it just puts a binary into a directory. It is simple, yes, not very extensible, yes. But it allows for clean and simple uninstallation. Updating is equivalent to "forcing" the installation.

With cargo install I actually started installing things into a per-user context because I trusted cargo: it just puts a binary into a directory. But if cargo install becomes extendible, it might put stuff anywhere in my home directory. And I'm not talking about malicious uses, but people who want to "do good" by instaling manpages or desktop shortcuts or file associations or similar.

If the RFC is merged in the current form, I'd probably stop using cargo install, and will be going back to patterns I've learned from C/C++: cargo build'ing and invoking binaries by their path inside the build directory.

Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

liigo commented on Mar 28, 2018

Maybe a more general way is post-build.rs, which can be used to invoke bindgen for example (not only for cargo install).

@est31

Instead I think we should focus on artifact generation: allowing post-build scripts to create .msi files or .snap files or .deb files and allowing users to install them on their own, or maybe even on the user's behalf.

I'm all for supporting that as well. I'd like to make packaging easier. (That also includes policy-compliant packaging that could go into distributions, not just creating a .deb from thin air without a source package.)

What I'm looking for here is a way to let crates provide additional files, not just binaries. Generating manpages, generating completions, and putting them in the appropriate places.

We'd be open to talking about how exactly that should work. For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation. That has downsides, in that it's a lot less flexible and puts more policy into Cargo, but it would support cargo uninstall.

Another possibility would be to just install those files into a temporary directory under the output directory, and cargo install would install everything under that temporary directory to the directory specified as its --root.

We'd like to do the simplest thing that could work, here.

@liigo We talked about having a post-build as well, but that doesn't solve the problem we're trying to solve. The issue is installing these files, not just generating them.

est31 commented on Mar 28, 2018

@joshtriplett

For instance, we certainly could have install.rs output a list of files to install and where they should go (including paths to files in the output directory that it just generated), and then cargo install could do the actual installation, and track the files for later uninstallation.

That would work as well. in fact we could have post-build.rs generate those files. In the traditional make install set up, the installation step is only copying stuff to the appropriate places, while the previous steps did the remainder of the work.

Your proposal (calling it "microcargo") sounds fine @joshtriplett but I'd like to give an alternative "monolithic Cargo" proposal to just be able to compare the two:

You could have post-build.rs output whether installation of the program is supported and if yes, outputs a list of the files together with a semantic information what they actually are: executables, manpages, completion files, etc... all need to be put into different places and some would require an additional set up step.

Cargo would then store that list and cargo install would just use that list as a recipe for installation. It would then decide based on input parameters whether to install on a global, or on a per-user basis, and depending on that it could then also tell users to set up a ~/.bash_completion or decide which of the completion files to install based on the used shell. It would then track all the files it installed and support uninstallation. Of course this would require to put some logic into Cargo, but after all, Cargo is not a generic make replacement but a tool specifically built for the Rust ecosystem. It does many opinionated choices already.

So with the "microcargo" approach you have install.rs parse all the input parameters and environment info (per-user or global, name of the shell, distro directory layout, which files the user wants to be installed) and output a list of desired destinations. The advantage is flexibility. The disadvantage is that writing a install.rs becomes far more complicated... I've seen so many issues with make install. Often this is not the main attention of the developer and so often it sucks in one way or another. With the monolithic approach, you would have a central place where improvement can happen. However, this downside of the minicargo approach would be fixed by the metabuild RFC.

So summarizing, compared to the "shellout" proposal of the current RFC, I think "minicargo" and the monolithic approaches are both better, with me preffering the minicargo approach.

Something that seems worth mentioning, explicitly: some responses have expressed the sentiment that the right answer is to work towards distribution packaging and similar, rather than making cargo install better. It seems worth pointing out, as someone who does care about distribution packaging as well, that making cargo install better will make it much easier to do distribution packaging. As mentioned in the RFC, we'd like to be able to cargo install to a directory and then package the result, and we'd like to make that include things like completions, manpages, and other associated files.

ashleygwilliams commented on Mar 29, 2018

So, as was mostly expected, the primary question brought up by the comments so far is the issue of uninstall. The current RFC presents the opinion that we should make no effort to offer an uninstall.rs, however it is an open question as to when an install.rs what we can do to mitigate the absence of an uninstall.rs. Several suggestions have been made, but I think that ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.

Would that appease any of those who are concerned about uninstall? If not, let's try to aggregate other suggestions about mitigating the uninstall issue and discuss which if any would be reasonable to implement alongside this, with an eye that there is still the goal of not packing cargo with so many features that it begins to resemble a platform-specific package manager. (which i think we all agree is something we DON'T want to happen.)

est31 commented on Mar 29, 2018

ensuring that cargo emits any files systems changes when an install.rs is run is a reasonable strategy to take.

I'm not sure I understand your proposal fully. Could you explain it? Suppose I did cargo install foo. With which command would I uninstall it? And how would it look implementation wise?

I think she's talking of a general class of solutions (correct me if I'm wrong), but one such proposal would be that install.rs emits cargo:installed-file:/path/to/manpage, and cargo keeps these strings around. On uninstall it removes these files and then removes the binary.

(This doesn't cover all cases of uninstall, but as the RFC mentions this isn't intended to be a generalized distribution solution)

@Manishearth Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

ashleygwilliams commented on Mar 29, 2018 •

@est31 yup, what both @Manishearth and @joshtriplett expansion of what i said is correct. i'm curious if that's enough to address the issue- and if not, i'm curious what strategies would best address the issue without growing cargo into a thing similar to a full fledged platform-specific package manager, which, i believe, we all agree that we don't want.

if we can agree on something i am very happy to update the RFC to include this strategy explicitly.

Centril commented on Mar 29, 2018

This strategy seems reasonable, and mostly assuages my concerns. There's of course the issue of regedit and such things on Windows where you are dealing with non-files. But I think that falls under not having a "full fledged platform-specific package manager"?

Personally I prefer the variant where cargo moves the actual files and the install.rs creates them and points to where they should be.

I think an argument in favor of this RFC is that we can always decide to permit an (optional) uninstall.rs file later on (if we wish / there's sufficient demand for it). And so we can start with a conservative initial feature.

One of the problems that crops up with other languages' package managers (hi, Python!) is that the tool chooses one particular convention for where to install files and uses it on every platform in the name of consistency. Usually this convention is highly-POSIX-flavoured, making things weird and awkward to use on Windows.

I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin, but is there a risk that adding more "install" support to Cargo will make it even more awkward on Windows, or might even make some tools unnecessarily Windows-incompatible if they try to install a file named "aux" or something?

ashleygwilliams commented on Mar 29, 2018

@Screwtapello so, i share this concern but i think this RFC alleviates it instead of exacerbating it. we specifically are picking to do a install.rs instead of a post-build type situation so that authors can detect platform and perform platform specific functions. there's no way to necessarily guarantee that authors will do it, but the motivation for this RFC, at least partially, is the fact that authors WANT a better cross-platform story.

aidanhs commented on Mar 29, 2018

I didn't see any mention of passing arguments to cargo install - is this a deliberate choice or just unspecified? If unspecified I'd like to propose allowing it (to avoid a world of ad-hoc environment variable usage).

ashleygwilliams commented on Mar 29, 2018

@aidanhs it was not opinionated! i think that would be fine, and likely uncontroversial? but i'm curious what others think (assuming no one objects i can specifically add that to the RFC)

est31 commented on Mar 29, 2018

the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things.

@ashleygwilliams

i'm curious if that's enough to address the issue

Yes, both variants as laid out by @joshtriplett would be enough (see quote above). I'd personally prefer the variant where install.rs is not doing the changes directly and would like install.rs to be sandboxed. It could have unrestricted read access but write access restricted to one directory inside the target/ hierarchy to ensure that install.rs is actually using cargo to install stuff.

Especially, it seems to me that the intention is not for install.rs to be as flexible as possible, as this would mean support for any kind of installation of applications (cargo replacing an OS package manager), but rather to allow the minimum of features for CLI applications to install their manpages and completion files.

ehuss commented on Mar 31, 2018

I realise that ship has kind of already sailed for Cargo, since it installs binaries into $CARGO_HOME/bin

@Screwtapello This may not be true in the future if something like rust-lang/cargo#5183 is merged.

Installing to a staging area, and from there copying to either ~/.cargo or /usr/local would seem sensible (depending on whether the install is global or user-local). Then the uninstall actions can be easily generated just by walking the staging area directory tree. Possibly custom install/uninstall commands could also be added for things like registry edits on Windows. Alternatively, the staging area could be a permanent one, with symlinks into one of the two aforementioned dirs. this is how Mac’s Homebrew does it.

I think that I'm against this RFC. Instead of allowing crate authors to put arbitrary steps inside of install.rs, I would prefer that we extend cargo and Cargo.toml to support additional crate artifacts types.

To make it concrete, I'll use man pages as an example. With this RFC, crate authors would write an install.rs script which would generate and move man pages into the correct location. I'll refer to this as the imperative approach.

An alternate approach would be to allow crate authors to put something like man-dir = "man" into their Cargo.toml, and let Cargo put any man pages it finds there in the appropriate place. I'll refer to this as the declarative approach.

I think that the declarative approach is superior for a number of reasons.

• Under the imperative approach, all crate authors must write their own install.rs scripts. Although they can certainly draw from each other's scripts, this is a great deal of duplicated effort. Additionally, installation is complicated by different distro and platform quirks and conventions, so many install.rs scripts will likely have bugs. With the declarative approach, this code lives in one place. This allows for less duplicated effort. And, when bugs are found, they can be fixed for everyone in one place.

• Different distros have different ideas about where things should go. For example, man pages will go somewhere entirely different on Ubuntu, homebrew, macports, nix, or windows. With the declarative approach, they can write a single adaptor script that parses Cargo.toml, which will work with all crates without further effort. With an imperative approach each distro will have to examine and adapt the install.rs script to identify man pages and put them into the correct location.

• With the declarative approach, we can add additional support for special handling of different kinds of crate artifacts. For example, we could support a command line flag to only install binaries and not man pages, or vice versa. It would be very hard to support this using the imperative approach, since all install.rs files would need to support this flag. Additionally, with the declarative approach, different crate artifact post-processing steps can be added as needed, for example converting to CRLF line endings on windows.

So, to sum up, I think we should extend cargo with the ability to understand and install crate artifacts on a type-by-type basis, so that they can be handled consistently and appropriately across platforms, and not left to individual crate authors.

@casey I think that's fair for a large number of cases, but what about certain Rust projects like tectonic, for which this RFC would be hugely useful in its present form?

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

@alexreg, what kinds of artifacts does tectonic need to install? I'm wondering if there might be a better way to support them.

Probably best we check with the Tectonic people, like @pkgw, since I'm not 100% sure at this point.

aidanhs commented on Apr 11, 2018

I could live with this being converted to an experimental RFC, or alternatively a nightly only feature that turns off sandboxing. But introducing sandboxing after the fact is quite a breaking change...

To echo others: robust sandboxing that supports Linux kernel 2.6.18, Windows 7 and OSX is hard (basically requiring a SUID helper binary on Linux). 'Good enough' sandboxing (assuming we agree on what that means) is still tough. Let's bear in mind that the goal of this RFC is modest - "make CLI apps easier to install". I'm sure we can come up with a way to make it so easy to do things the 'right way' that most authors never even think of anything else, or just explicitly document that anything you do that isn't creating a file in a specific directory may break in the future.

I like @RalfJung's take:

I like to think that the reason make uninstall is broken is not that software authors want to make the lives of people that uninstall their software miserable, I'd like to think that they actually would prefer uninstallation to work so if we tell them "that's easy, just use the wrappers in this lib for installation and you are all set", they'll just use it. I hope I am not being too naive here :D

Minor: the discussion seems unix-centric - isn't regedit is a fairly important part of CLI app install on Windows (e.g. to put it on your path / to identify where your app is installed to find assets)? But files on disk is step 1 (and devs can perform 'post-install' steps in their binary) so maybe it's ok.

Well, my above suggestion was to use some convention based (Unix-like) FHS-like approach (@jan-hudec expanded on this with good ideas). For registry edits, I suggested we just have install.rs scripts write directive to stdout, which get interpreted by cargo. They'd be ignored on all platforms but Windows (possibly with warnings output, since ideally you'd want install.rs script authors to check the OS before emitting them).

One point of discussion is "who sets conventions - ecosystem or cargo". The sandboxing thing (to me) ends up implying "cargo decides conventions" (since you're only empowered to make decisions on things cargo says you can). I'm ok with collectively selecting that as the approach, just wanted to point out that these are linked.

Makes sense to me.

est31 commented on Apr 12, 2018

One point of discussion is "who sets conventions - ecosystem or cargo". The sandboxing thing (to me) ends up implying "cargo decides conventions" (since you're only empowered to make decisions on things cargo says you can). I'm ok with collectively selecting that as the approach, just wanted to point out that these are linked.
[...]
I personally just want curl ... | sh rewritten in Rust

I'd say that the approach is already "cargo decides conventions".

Make is basically a glorified shell... for the execution of targets, make actually invokes a shell, the syntax is similar, etc etc.
CMake is no different. It has some special syntax built in but every library did its own thing and reimplemented the same stuff over and over and it is still an ugly mess.

Cargo's traditional approach is radically different. Instead of being a generic tool that allows everything, it has a ton of stuff hardcoded that the code has no say over.

• it only supports Rust
• non-git non-path deps are hardcoded to the crates.io domain (at least unless the user changes .cargo/config)
• it has the concepts of build, run, and test

When I came to Rust, it was surprising to me to see all of this stuff hardcoded. But then I saw how all of this hardcoding is benefitting the Rust ecosystem.
You don't have non-rust stuff to worry about, you don't have everything spread out on several hosts with possibly several policies, you know instantly how to compile something (just do cargo build).

So I think we should follow that approach and be opinionated for the user's sake :).

To echo others: robust sandboxing that supports Linux kernel 2.6.18, Windows 7 and OSX is hard (basically requiring a SUID helper binary on Linux). 'Good enough' sandboxing (assuming we agree on what that means) is still tough.

It doesn't have to be watertight. I think something that causes big headaches for people who ignore the convention (or who don't know... most times it is negligence :p ) is enough. I don't want to go into detail about implementation but I think the two options on the table are gaol and interpreting stuff with miri.

Any progress on this lately?

I suppose this hasn't been high on people's priority lists lately, given the lack of news. Any chance of reviving things?

Some questions I had:

• Is the plan now to make cargo closer to a lightweight package manager, per above discussions?
• Should we retain separate "repositories" of Cargo binaries & their support files that switch depending in the toolchain, in a pyenv-like manner?
• What's the interaction with Cargo components going to look like in the future? It seems like components could just be "official" versions of Cargo packages, if we extend the cargo install system to support a more general concept of "packages".

derekdreery commented on Jul 25, 2018 •

It makes the emergence of conventions significantly more difficult as the option to reuse, share, or automate this task has significantly affordance than integrating it into the familiar cargo step.

I'm struggling to parse this. Can you tell me what you mean?

cargo install could optionally have a flag to disable the running of the install.rs file.

What's the use case for this?

One of my use cases of this feature is to enforce all of the collaborators to set up git-hooks scripts to enforce some policy like commit message and others. I usually use husky to enforce all collaborators to follow conventionalcommits

Looks like this is being revived as part of rust-lang/crates-io-cargo-teams#14. @ashleygwilliams / anyone else, how are we going to proceed here?

This RFC sounds excellent! In particular, this approach to offering cargo uninstall strikes me as very reasonable:

....Right, the idea would be that install.rs either prints a list of all filesystem changes it wants to make but doesn't make them directly, or alternatively that install.rs only installs things to a subdirectory of the output directory and then cargo install installs those things. Either of those would allow cargo install to optionally remember what it installed for later uninstallation.

I was surprised to see that there hasn't been any movement on this in a year and a half. Has it been superseded by more recent initiatives? If so, could someone provide a link?

joshtriplett commented 19 days ago

This came up in today's @rust-lang/cargo meeting. This hasn't seen activity in a long time, and as a result, we'd like to postpone it. This should not be taken as a lack of interest; we just don't want to leave this open and unresolved indefinitely. If someone would like to pick this up, they should feel free to do so, and post a new RFC.

In general, if someone is interested in reviving this, they should specifically write this in terms of being able to stage things in an output directory and letting Cargo install them, rather than installing things directly. We're generally not in favor of a completely general "install things directly onto the filesystem" script. (We're also uncertain about the amount of complexity this might create in Cargo, and we'd want to know this wouldn't impact long-term maintainability.)

Hopefully this guidance helps someone looking to work on this in the future.

@rfcbot postpone

rfcbot commented 19 days ago •

Team member @joshtriplett has proposed to postpone this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

rfcbot commented 19 days ago

This is now entering its final comment period, as per the review above.

rfcbot commented 9 days ago

The final comment period, with a disposition to postpone, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC is now postponed.