Rust 2019: Address the “Big” Problem

 4 years ago
source link: https://www.tuicool.com/articles/hit/FzyYJ3j
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

samsieber 2018-12-27 19:10:18 UTC #1

Tame Complexity through Community Involvement Tools

Edit: I’d just like to say that I think 2018 has been a great year for rust. Rereading this, I realize may have come off as a little negative. End edit.

Rust has a “big” problem. Our feedback loops have gotten big because we have gotten big (as a community). We’ve started council together at the end of the year. I don’t think it’s bad, but part of it is reaching consensus on the current state of Rust. I think it’s good actually, but we need more things like it - I’m not suggesting we call for even more blog posts, but that we start tailoring activities to address the “big” problem.

The “big” problem is growth (and increased community size) - it’s a good problem to have, but it does cause issues.

Rust has a lot of gaps caused by it’s explosive growth, more prominent ones have already been covered:

These are all artifacts of growth. It’s gotten harder to hold everything about Rust in your head. The tools the Rust community has been using (mostly github provided ones) were good at small sizes, but don’t scale well. They’re leaky abstractions. Pros and Cons analysis with consensus represented by a linear discussion, state scattered across the web, etc. Harder to follow, harder to moderate, harder to contribute to.

It’s harder to:

  • track things you care about (I see this as a large contributing factor in the impl Trait return position & website controversies)
  • see the overall status of rust
  • see where things need help


We should build something to fix that.

We should build Rust specific infrastructure to manage the complexity that comes community sprawl (the sprawl is not bad). Organization specific infra is a thing (I’m looking at you task-cluster that replaced jenkins )

What will it look like? I’m not sure, but I suspect each team will need something different - form following domain.

I have a couple of ideas after reading some other Rust2019 posts of what said central, easy to find tooling should include:

RFC Stuff

We should probably get an RFC tracker. A real tracker, not github issues (Can you imagine if https://caniuse.com/ used Github issues/prs for it’s UI? ). Lets track by time and state, category and syntax impact (and more?!). There is so much metadata about RFCs that would be useful if it was understandable by computers. Lay people could get a better idea of what’s going on, and contributors could see what’s happening at a glance.

In addition to the overview/tracking tool, we need a better tool for contributors of RFCs to use. Discussion is the big one. Can we build a tool for structured discussions - pros/cons, links to other RFCs, view changes overtime and hidden votes (because while things aren’t a popularity, it’s probably still a good idea to let people express themselves, even if only so the moderators / leaders can see). There are also proposals forlimiting throughput, a more distinct staged process , needing a team champion, etc. We could integrate that too.


I heard the docs need love. So it’d be great to have a list of things that need doing. Sections to proofread, sections that need feedback (e.g. I’m trying to accomplish x, but it’s not working / coming off right), things to write (cookbook), brainstorming structure/approach ideas, etc.

It’d also be nice to a have one big index to rule them all, about the various docs, what’s next on the doc teams plate (we’re hoping to improve x, other big initiatives, etc), etc.


The embededded working group could probably use a page listing all the platforms, and what their status is, along with a list of drivers. It actually probably exists. Somewhere.

There are probably other things for other teams that would be great to put into this one, mythical centralized location. An improved team index would be nice (e.g. direct links to the discord / other place of preferred chat, links to the team specific areas, etc.)

If we were to open infrastructure support to contributors (I have a perfectly good machine laying around I’d be willing to run a CI runner on 24/7), we’d probably want a public dashboard of some sort listing current capacity and a way to volunteer your computer. Barring that, we’d probably want a graph of average wait time on compiler PRs (which is measured in days!).

There is surely a lot more than this to what we’d want (I think) - I’m not really familiar enough to say though. Heck half these things are from what I’ve read in the 2018 posts, if not more.

In Closing

The rust community has been great. In addition to the great language, the people are great, and trying hard things - transparent consensus (well, taking in put into account)-based decision making. It’s been harder to do that at scale (it either exacts a big toll or breaks down). I think we should fix that.

This is a call to trade generic tools for hand-crafted tools. To trade in a little team personality (each team does stuff differently), for less friction (and more accessible to new people).

Custom tooling is a big effort. But I think it’s the best way a growing community long term without giving up the great things that characterize Rust’s governance and growth.

kennytm 2018-12-27 20:03:19 UTC #2

FYI a graph like this is available in https://rust-lang-nursery.github.io/rustc-pr-tracking/ . You could find this link fromforge.rust-lang.org.

samsieber 2018-12-27 20:19:26 UTC #3

Those are nifty graphs. I’ll take those QZrMbi3.png!web

I think that in terms of gathering support for expanding infrastructure, a better graph would be of the average wait time of things that are S-waiting-on-bors (and perhaps S-waiting-on-crater). That would really drive home the infrastructure woes. The other graphs are useful as well though because they highlight the people component (review/author/team/bikeshed). I wouldn’t want to lose any of the current graphs.

anp 2018-12-27 21:08:48 UTC #4

As someone who’s built a few pieces of custom tooling for Rust including rfcbot and a now-defunct dashboard with many of the metrics you’re describing, IMO the bottleneck on these ideas is availability and prioritization of effort/time/labor/work/etc. This kind of work doesn’t typically carry lots of prestige (one of the main motivators for contributions), often requires lots of information gathering if you’re not building tooling for your own processes, and is often hard to prioritize against other “direct impact” work. To build things like those described, we either need to find people already in the community with the right desires/skills/experience/etc, attract people to the community who would like to contribute to projects like these, or fund their design and construction via someone’s full-time work.

The first option seems to be hard because existing contributors are often furiously working away on the topics which originally captured their attention. I can see how some existing proposals to “do less” might free individuals up for some meta-work, most proposals I’ve seen have other proposals for priorities that those contributors might adopt to pay down organizational debt within each of the teams. There have been a few attempts in the past to rally contributions around these kinds of custom infra applications (the bors2 conversations in the community team a little while ago springs to mind) and I at least am not aware of any of them taking off. A new effort in that direction would, I hope, address typical pitfalls for coordinating that kind of work.

Aside: which team would lead the design of an RFC tracker? Core? Infra? A new one? Questions about the governance of tools like the ones proposed have already come up for rfcbot and figuring out some answer for their home seems like a good idea.

The second option to get people working on these is an interesting one but still reduces to a pretty similar problem: who among the existing community will lead the design and implementation of the new tools? What tools will they have to succeed?

In terms of full-time funding, I’m not sure who would fund this work – Mozilla? They seem a bit capped out on the Rust initiatives they’re funding (although I don’t have any special info there). Some large tech companies have expressed interest in funding work on Rust, but AFAICT they’re mostly focused on specific things they need to help it succeed in their environments. Perhaps there’s another angle to play there?

EDIT: I worry that I enumerated a bunch of obstacles without expressing enough enthusiasm – this is a great set of ideas and some combination of them should definitely happen! We’ve talked about trying to make rfcbot into various forms of this for a little while (@nikomatsakis and I talked about a more general-purpose RFC tracker and workflow organizer at rustconf…2016?) and have been mostly blocked on my time and a slow trickle of contributions. Arguably rfcbot could have been the wrong choice for where to build this (the code isn’t exactly polished lol), arguably my lack of time has meant that mentoring new contributors hasn’t happened when it could have, etc. Not saying that any of this is impossible but that the lack of these tools today seems to me more due to the underlying question of Who? than whether they’d be desirable.

samsieber 2018-12-27 22:02:49 UTC #5

I think we’ll end up with a couple of different groups of contributors in this:

  • Disgruntled people who have a hard time following things, who want to see it better
  • People who finish whatever they’ve been enthusiastically doing (either completing or burning out/turning over the reigns)
  • People who love stats & transparency IfYRnmY.png!web … maybe.

I’d think at the very least we’d want a working group for this, probably with people from each team at least for coordination. Maybe a proper team? It seems like it’d be the in the purview of both Core and Infra, and yet maybe a big enough undertaking to be separate.

I didn’t even think about funding, but that’s a really good point - it will take a lot of bandwidth. I do think that the easiest thing to secure funding for would be for more infrastructure (if we have nice graphs and projections showing potential impact of contributions on turn-around time).

It’s funny that this is meant to be a way to help solve coordination issues, and the biggest threat to its success is probably coordination issues.

I think the following will be pivotal to getting things off the ground and successful:

  • getting a small core estabilished - sort of like how tower-web and warp were worked on for a bit before being released. It’s important to get a core established to have something to fit the pieces of the full implementation into. And we don’t want a bunch of bike-shedding to prevent the groundwork from happening.
  • Having a coordinator. Once we get far enough, I imagine that there will be lots of little things that could be worked on in parallel. Maybe not though.
  • Emphasis on a couple of highly visible things for the core - PR wait-time and RFC tracking perhaps? Something that’s not too hard, and yet is would be liked by a lot of people. I think average turn-around time and RFC tracking brings a lot of simple to see transparency and benefit
  • Official backing - namely a place to live (both domain and server wise), and guidance in terms of extracting / integrating with the current crop of rust … stuff. Data sources and points of interaction.

A large part of me thinks we’d do well to make it modular so other things could reuse it as necessary. Sort of a code level modular Drupal / CMS thing? (disclaimer: I’ve never touched PHP/Drupal). In that linked thread you can see that the RFCBot is being used outside of Rust.

Anyway, I think these are good points that need to be brought up and figured out - thanks for sharing them. If I had to choose between hearing about obstacles and hearing only enthusiasm, I’d choose to hear the obstacles any day QZrMbi3.png!web .

Hopefully putting an emphasis and discussion such tooling will help get the “who” question answered.

flavius 2019-01-01 09:10:22 UTC #6

Is forge linked to from the main site? Writing from mobile.

llogiq 2019-01-01 09:15:34 UTC #7

Regarding RFCs, I’d like to see a collaborative mind-mapping tool where people could “upvote” branches to make them appear stronger, and hide (or, for mod team members, even remove) them. This could be a great help to map out the design space.

Regarding docs, I held the docs workshop at the RustFest 2019 and I think doing more of such workshops off- and online would be a good start to improve matters.

madmalik 2019-01-01 11:55:47 UTC #8

I really like the idea of rust specific infrastructure, mostly because i was thinking about it too ARzuqin.png!web

I’d volunteer time for that and i actually view it as high impact work, at least in the long run.

I also prefer starting with a very small scope and slowly build features out from there, feeling out what is really needed and accepted by the rust community while were going.

But without fulltime work, this is a really big project. It’d be a big project for a fulltime team…

zoul 2019-01-01 15:40:52 UTC #9

Exactly. Is it completely naïve to think about pooling resources with other languages such as Swift? After all, the problems being solved are very similar, both in the domain and in the social dynamics. I just don’t know if the required coordination wouldn’t outweight the advantages.

Soni 2019-01-01 15:50:43 UTC #10

I think git is a suitable tool for tracking PR discussion.

About Joyk

Aggregate valuable and interesting links.
Joyk means Joy of geeK