Mozilla could remove user.js from Firefox in the future

Tracking

Status:

NEW

People

(Reporter: Gijs, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: main-thread-io, perf, Whiteboard: [fxperf:p2] [fxperfsize:S])

(In reply to Mike Kaply [:mkaply] fromcomment #1)

Anecdotally, the reason people use it is because it gives them a really easy way to maintain preferences when they reinstall/move/change Firefox.
I think we actually have telemetry on usage.

Can you point me to that telemetry? I couldn't find it from a quick look...

(In reply to Mike Kaply [:mkaply] fromcomment #4)

https://searchfox.org/mozilla-central/source/modules/libpref/Preferences.cpp#4032

Telemetry::ScalarSet(Telemetry::ScalarID::PREFERENCES_READ_USER_JS, true);

I've never looked at it in telemetry, I just knew it was set.

Ah, whereas I looked at telemetry.m.o, and it expired in 62 so it's not in there for current nightly...

Anyway, 2.5% of pings in nightly 61, 2.7% of pings in beta 61. Of course, that still doesn't tell us who put it there, what's in it, and if users actually want it...

Seems likely to be search hijacking, given that the search service uses the default branch to read preferences that the user isn't expected to tweak, eg. defaultenginename (the original default engine that we use as a fallback if all else fails), geoSpecificDefaults.url (the url of the absearch server),

defaultenginename is only read if there is a distribution. We've never seen this used for hijacking.

user.js has been around for over 15 years and people have known about it for that long. There are articles everywhere about it. ghacks posts custom user.js files. There's tooling that creates user.js files - http://configfox.sourceforge.net/ .

Your assumption that it is only used for nefarious purposes is misplaced. Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

Just because you don't see a use for user.js doesn't mean other people don't. Firefox is a very configurable browser. We shouldn't be surprised when people configure it.

(In reply to Mike Kaply [:mkaply] fromcomment #10)

Seems likely to be search hijacking, given that the search service uses the default branch to read preferences that the user isn't expected to tweak, eg. defaultenginename (the original default engine that we use as a fallback if all else fails), geoSpecificDefaults.url (the url of the absearch server),

News to me, but OK. about:config and prefs.js, yes, but user.js ?

ghacks posts custom user.js files.

1. ghacks are very niche. This isn't anywhere near sufficient to explain 3% of pings.
2. They seem to have stopped doing that in 2017.

There's tooling that creates user.js files - http://configfox.sourceforge.net/ .

That (based on the changelog) hasn't been updated since 2015, since before photon and the last supported ESR.

Your assumption that it is only used for nefarious purposes is misplaced.

My main assumption is that 3% of users shouldn't want/need this thing, and 97% of users shouldn't be paying startup costs for it.

Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

Which are what? I'd expect admins to use the other default pref locations for changing defaults, as user.js is per-profile. What value does user.js provide to them / why can't they use the app-wide location?

Just because you don't see a use for user.js doesn't mean other people don't.

Come on, don't patronize me. I've been around here long enough to know this...

Firefox is a very configurable browser. We shouldn't be surprised when people configure it.

This is a strawman. I'm not objecting or surprised people configure Firefox. I'm surprised that they would use this file to do it.

It's reasonable to be surprised when 30-300x more people apparently create a config file in a hard-to-find folder for which there is no UI, no documentation (only on "here's a list of reasons things might not work properly" pages on SUMO), no prompting, than use some browser features like feed support for which we did have UI, the removal of which was described by some as the end of the bloody world. Same ratio with master password use, (0.11% of beta pings). 3-4x more users than people use any of the URL bar page action buttons (this includes bookmark, pocket, anything they customized there themselves, add-ons in the URL bar itself).

The stats here are surprising -- and also uncertain, because the probe only records pings where the file exists, not pings where it doesn't, so I'm forced to compare with other pings (I used GC_MS to get the 3% number).

In any case, it seems your argument is there is a usecase for this file. I'm aware of the one (other) Mike described incomment 6 - is there something else I'm missing? Besides the reasons Mike cited, and besides "setting the machine-wide defaults" for which you could use the app locations, why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

Well the advantage here is that you can have a file that you keep around and just drop into a profile directory and Firefox doesn't mess with it. That's what I've heard anecdotally.

I'm fine with switching this to a pref that's read to then read user.js. That makes sure that only people that want to use it use it.

I think the only problem there would be that if some preferences had already been user set to default values that then get set by user.js, there might be an issue.

We could also do some user surveying (similar to what was done around userContent.css/userChrome.css) to figure out if there is a usecase we're missing.

And I agree with you 100%. People complain loudly before removing features and then when we do remove them, there is just a whimper.

Maybe the secret is to start with only a browser and a URL bar and add features when they requested. Then we're never stuck with features people don't want :)

To give you another reason why some people use the file user.js: I can't speak for support in other languages but in German-speaking Firefox support we often recommend to use user.js instead of about:config as best practice and Gijs wrote the main reason, probably without knowing that he wrote about a good reason to use user.js: :P

(In reply to :Gijs (he/him) fromcomment #8)

Once things break it's a nightmare to figure out /why/ things are broken, because generally they just copied some instructions off a random webpage 3 years ago and forgot all about doing that, they just wonder why a particular Firefox widget is fubared and either switch to Chrome/Safari/Edge/whatever or cause us / sumo / mozillazine to spend ages debugging for a problem that, fundamentally, shouldn't exist.

That's exactly what happens if people apply changes via about:config: they forget that they changed things and what they changed. In the user.js file people can add notes what the options do. So people a) find all their changed options (which is, by the way, much more difficult with the new about:config!) and b) they can have notes about all their changes.

Of course, with user.js people still can forget that they changed things and what the options do. But wihthout user.js people will forget (almost) for sure.

Bonus point: It's very easy to apply changes to more than one system. Since about:config does not have an export/import feature it's a really nice feature for people who prefer to have the same settings on all their systems.

I hope this comment is helpful.

(In reply to Sören Hentzschel fromcomment #15)

Bonus point: It's very easy to apply changes to more than one system. Since about:config does not have an export/import feature it's a really nice feature for people who prefer to have the same settings on all their systems.

The solution for import/export is generally considered to be "use sync"; there's a separate enhancement request for export/import for prefs, I'm pretty sure. But IMHO we have already lost if people routinely have to resort to setting prefs in about:config or similar.

I hope this comment is helpful.

It is - it helps me understand why the German support forum might suggest this to people, but for the sake of the German support forum I hope you're not having to tell 3% of our users this, as we should then probably hire fulltime people to work German support given the volume of users that implies.

What do you tell people to change in user.js in this way anyway ? Is it really routine that people have to change prefs that aren't in the UI? Which ones? That in itself is pretty concerning tbh...

Anyway, the problem is that none of our support or engineering system is structured for detecting that this has happened (apart from, apparently, the German support forum). Some prefs, as already said, will break things if users change the default value. about:config won't say that prefs are modified if modified through user.js . about:support doesn't tell you which prefs are set, either, just whether the file exists (also, I just foundbug 1544573 which makes this even more fun).

I assume that the support forum can deal with having to set a pref in prefs.js in order to enable user.js, or using the app dir version instead ? (ni for this)

Longterm, I'd really like to evaluate whether we can remove support for this file entirely, because it just fundamentally doesn't really make sense to have so many different files that all control the same thing, but it probably requires figuring out why so many people use it, which we don't have cycles to do. Nor is it really obvious how we'd go about doing so: if we think a substantial portion of people aren't aware they've done this, just doing a survey "why do you have this file" is unlikely to be enlightening; we could try doing telemetry on what prefs get set, but we'd probably have to have some kind of strict list of prefs we allow ourselves to send back to avoid passing back user data, which again might not get us the data we need.

(In reply to Mike Kaply [:mkaply] fromcomment #14)

I'm fine with switching this to a pref that's read to then read user.js. That makes sure that only people that want to use it use it.
I think the only problem there would be that if some preferences had already been user set to default values that then get set by user.js, there might be an issue.

I'm confident this works because it's more or less what Normandy does today. You can write to the default pref branch and it'll just overwrite the previous value. We read prefs.js and user.js one (immediately) after the other today based on what I'm seeing in the performance profile IO markers, and given one gets read into the default branch and one into the user branch, that should continue to Just Work.

(In reply to :Gijs (he/him) fromcomment #16)

What do you tell people to change in user.js in this way anyway ? Is it really routine that people have to change prefs that aren't in the UI? Which ones? That in itself is pretty concerning tbh...

Totally different because people are so different. ;-) browser.tabs.loadBookmarksInTabs is one example for a change that is "frequently" asked for (not frequent at all in relation to the total user base but it's more frequent than other requests). There is no visible setting so we have to tell the users that they have to use about:config or user.js. browser.tabs.tabMinWidth is another example.

But it's not always that we ask people to change a hidden setting. People report problems because they forgot that they changed something in the past. Popular examples are javascript.enabled and privacy.resistFingerprinting. People read articles on websites which recommend settings like privacy.resistFingerprinting because of some privacy improvements but these articles don't say anything about the side effects. People change these settings and a few months later they need help because something is broken like the installation of add-ons on addons.mozilla.org. In such situations we tell the users about the advantages of user.js (have an overview about all changed hidden settings and the ability to make comments).

(In reply to :Gijs (he/him) fromcomment #16)

I assume that the support forum can deal with having to set a pref in prefs.js in order to enable user.js, or using the app dir version instead ? (ni for this)

Yes, I don't see a big problem with that.

The priority flag is not set for this bug.

:njn, could you have a look please?

For more information, please visit auto_nag documentation .

(In reply to Mike Hommey [:glandium] fromcomment #21)

180ms to open a non-existing file seems ridiculous, especially compared to other I/O happening earlier.

I/O time is random. It's typical that some I/O calls will take <1ms and the next one will block for more than 100ms. We even sometimes see this when opening several times the same file. Eg. in this same startup profile https://perfht.ml/2T6mH6O , opening C:\Program Files\Firefox Nightly\fonts\TwemojiMozilla.ttf takes 50ms on the parent process, and a few seconds later 606ms on the WebExtensions process.

I don't know what the profiler is supposed to be reporting there, but it seems fishy that there's only 1 sample for 180ms spent there.

Significant gaps in the sampling usually indicate that the machine is CPU starved. Which is pretty common on low end hardware.

On a profile with an interval of 1ms, I'd expect 180.

On Windows the minimum sampling rate currently supported is 2ms. So even when the sampling is configured at a 1ms interval, you would get at most 90 samples during a 180ms period on Windows. Even if the machine had plenty of spare CPU cycles available.

At this point, I'm starting to doubt what the profiler is reporting for I/O, and I'd be curious to know what e.g. Xperf would say happened during the same interval of time.

I/O interposition causes significant overhead on Windows, but that overhead is after the I/O markers, when trying to find the filename associated with the file handle. This overhead is because the HandletoFilename function itself triggers I/O. Which again takes a random potentially long time. In the same profile, we can see some of this overhead here: https://perfht.ml/2OCA7J8

(In reply to Florian Quèze [:florian] fromcomment #22)

180ms to open a non-existing file seems ridiculous, especially compared to other I/O happening earlier.

Is that your conclusion from observing profiles from the Gecko profiler, or with something else like Xperf? If there is something fishy in the Gecko profiler, you have some sort of case of confirmation bias occurring.

(In reply to :Gijs (he/him) fromcomment #25)

You could write a prefs.js file with the pref to tell it "read the user.js file" as well, right? (In fact, you could probably just put all the prefs in prefs.js directly if it's a new profile...)

True, thanks for that. We could make a workaround for our project, but it does affect other consumers of mozprofile. I suppose the maintainers of that library could account for this and do what you suggest automatically, thus enabling current use cases.

ghacks posts custom user.js files.  ...  They seem to have stopped doing that in 2017

It's not true.

(In reply to :Gijs (he/him) fromcomment #12)

(In reply to Mike Kaply [:mkaply] fromcomment #10)

ghacks posts custom user.js files.

this was already mentioned, but i want to mention again that the ghacks config was moved to github long ago and has been a very active project ever since - how many people use it, i don't know, but i suspect the number is higher than you imagine - the ghacks repo also includes an update script to fetch the newest user.js

i've written several technical articles regarding Firefox and the 2 general config articles, both of which rely heavily on the ghacks user.js, are by far the most popular articles on my website for years (and they are kept updated)

Your assumption that it is only used for nefarious purposes is misplaced.

97% of users aren't using a user.js, so how is there a startup cost for them?

Just a couple weeks ago I was at a conference for Mac administrators and the subject of user.js came up. Lots of people use it for regular purposes.

when multiple users are sharing an installation?

It's reasonable to be surprised when 30-300x more people apparently create a config file in a hard-to-find folder for which there is no UI, no documentation ...

there's docs all over, including on mozilla.org

... why would you want to change the default value on a per-profile basis instead of changing prefs.js / about:config ?

a, so you guys can't change it back (no offense) and b, simplicity - admins probably aren't using this file as you imagine, but lots of general home users probably are and the user.js makes pref management easy, portable, and it survives changes to prefs.js

and using about:config is a horrible way to incorporate custom prefs for those that know what they're doing, and perhaps in general - the only reason i ever suggest using about:config is to test something and then revert it back

i suggest taking a look at the [ https://github.com/ghacksuserjs/ghacks-user.js](ghacks repo) which i suspect is the most popular distributor of a custom user.js, at least in the U.S.

as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored

frankly i think the suggestion to ignore user.js is a monumentally bad idea

I totally concur with @bugzee, esp. with:

"as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored

frankly i think the suggestion to ignore user.js is a monumentally bad idea"

Configuration of FF is its strongest point. IMO, making that harder or impossible to do (which has already happened quite a bit lately) is shooting yourselves in the foot, IMO.

The solution for import/export is generally considered to be "use sync"; there's a separate enhancement request for export/import for prefs, I'm pretty sure.

Until then, there's no way to export the modified preferences. There's not even a way to list them, and in any case most of them are changed automatically by the browser. Do I want to export devtools.debugger.pending-selected-location , datareporting.policy.dataSubmissionPolicyNotifiedTime , or browser.urlbar.placeholderName ? I have no idea what they are. But I know what's in my user.js file.

And of course, there will be people that won't want (or be able to) use Sync. Or say that I use Sync for my changed preferences. What do I do if one of them causes a problem a couple of years down the line? How do I even know it's there?

But IMHO we have already lost if people routinely have to resort to setting prefs in about:config or similar.

Perhaps we should remove support for about:config , then? Just so people don't break their profiles when digging through it? With user.js I only have to keep a file in sync across all my PCs. Should I make a list of the settings I need to toggle in the UI? Are all of those preferences exposed there?

(In reply to bugzee fromcomment #28)

My main assumption is that 3% of users shouldn't want/need this thing, and 97% of users shouldn't be paying startup costs for it.

The cost is touching the disk to check if the user.js file exists on the disk.

as for the this ticket, as a user i'm semi-ok with having a pref to tell the browser to read user.js, but i think you're gonna anger a lot of people who have a user.js for good reason (privacy, security) and they find out at some point down the road that it's being ignored

We could land the changes for this bug in two steps, with one release checking for user.js and creating the pref if a user.js file was found, and the future releases after that only touching the disk for user.js if the pref is set. This is how we handled the same change for userContent.css inbug 1541233 withbug 1550157 that landed first.

Please stop this.

I have been using netscape navigator / firefox for decades and user.js has certainly been around > 15 years.

I am really getting tired of doing enterprise and personal support of firefox and dealing with changes that completely break things because someone makes a claim that startup time is critical, or doesn't see a use-case for themselves:

(per Kaply's comment that you don't hear users loudly complaining post-change, perhaps it's because people are giving up and leaving. My frustration level with all the recent breakages with profiles and Policies not supporting things like "defaultPrefs()" and my feeling that all the stuff that worked great for the past decade (autoconfig/mozilla.cfg/defaultPref/lockPref/...) is going to be replaced with something less functional is very high)

$ strace -f -e trace=stat -c firefox -new-instance -profilemanager

% time seconds usecs/call calls errors syscall

100.00 0.011346 5 2205 1548 stat

100.00 0.011346 2205 1548 total

(IGNORING the 40 subprocesses spun off during that interval that i'm sure are consuming a lot more wallclock-time)

Please tell me how ONE stat() for a missing file is gonna make any real difference here. Same for the removal of processing user{Chrome|Content}.css.

prefs.js is NOT an answer here, as the entire point behind user.js is to do something before and independent of prefs.js. prefs.js is volatile, user.js is not. (user.js is also documentable).

There have been plenty of justifications/use-case arguments here for why to use user.js, and i don't have much more to add, other than i see a lot of single-user-centric thought going on in firefox development these days that ignores how enterprises and workgroups and folks with multiple machines/location who don't want to buy into 'sync' use it.

I want to create useful defaults for users at the enterprise level, but give them the freedom to alter those settings if they aren't done for security policy reasons (lockPref()). Allowing a user to create a user.js is one of the best ways to do this. My users do NOT have access to the APPDIR/INSTDIR (at least on Linux, nor if they don't have ADMIN rights on Windows/MacOS). so how can they modify that? they can't. but they can modify ~/.../user.js. (and no, letting users install their own copies of firefox in their homedirectories is not an answer -- that's another security debacle)

Also, as someone else pointed out Import/Export of config for exchange to prefs.js (or such) is much more likely to cause problems because the entire configuration space is being dumped. Deprecated settings will persist, whereas user.js will presumably have only preferences applicable to the needs of the user.

As for telemetry... If i turn it off/opt-out -- it's likely because i want customization, so you aren't weighing my desires for such customization.

thanks for your consideration.

--stephen

(In reply to sdowdy fromcomment #32)

$ strace -f -e trace=stat -c firefox -new-instance -profilemanager
% time     seconds  usecs/call     calls    errors syscall                                                                                                                                            
------ ----------- ----------- --------- --------- ----------------                                                                                                                                   
100.00    0.011346           5      2205      1548 stat                                                                                                                                               
------ ----------- ----------- --------- --------- ----------------                                                                                                                                   
100.00    0.011346                  2205      1548 total

Is this just for profile manager and then quitting? Because that's certainly not expected - I'd expect a fraction of that number of stats, and 0 subprocesses, maybe 1 on Windows (which this presumably isn't given strace ) for the launcher process. Please file a separate bug so we can investigate. Details about what these stats are would help.

(In reply to :Gijs (he/him) fromcomment #35)

Please file a separate bug with more details.

I just filedbug 1579607 (and left it untriaged, as I don't know what it should be under).

Please remove the needinfo? from [email protected] , as that should probably have been me, but it's not me.

(In reply to Laurentiu Nicola fromcomment #36)

Please file a separate bug with more details.

Thanks, will take a look.

Please remove the needinfo? from [email protected] , as that should probably have been me, but it's not me.

D'oh. Thanks for pointing this out. Going to leave the ni for sdowdy for now, as I suppose there's no guarantee that the stats they're seeing are the same ones you're reporting...

(In reply to sdowdy from comment #38)

(In reply to :Gijs (he/him) from comment #33)

(In reply to sdowdy from comment #32) ...d expect a fraction of that number of stats, and 0 subprocesses, maybe 1 on Windows (which this presumably isn't given strace) for the launcher process. Please file a separate bug so we can investigate. Details about what these stats are would help.

Hmm, i can't reply via e-mail to the needinfo request, it seemed to blackhole (i got two messages, one said do-not-reply, i replied to the other that didn't say that

Yeah, email reply on bmo is not a thing, never has been AFAIK.

that's JUST for spinning up profile-manager and exiting [EXIT] (w/o creating a profile)

Thanks for clarifying this, and the follow-up investigation. I will be poking at some of these and filing follow-up bugs where appropriate. Just to briefly respond to some of your points:

My point is those many stat()s *(successful and not) happen regardless, so 1 stat() isn't going to make much difference.

At least on Windows, this is not the case - we see such stats taking literally >100ms on slower (non-SSD) machines. In terms of Firefox startup time, that's a lot of time that could be spent more productively. Similarly, we care a lot more about startup IO than about IO that happens "later". That is, ideally Firefox should do the absolute minimum of IO prior to showing you a window and loading the homepage / restored primary tab's page. AFAICT, the strace metrics also do not account for which threads do the IO. Doing background reads is a lot less problematic - I'd expect the sqlite ones to fall under those, for instance (ie I expect them to not block the main Firefox process starting up), and in fact they are likely to be happening in the child process, rather than the main process, too. Of course, there could be disk contention, but so far this seems much less of a problem than the main thread simply being stuck waiting for IO (certainly on slower machines). There's already a list in https://searchfox.org/mozilla-central/source/browser/base/content/test/performance/browser_startup_mainthreadio.js of IO that we know happens on the main thread when Firefox starts, including data about which platforms it happens on and during which phase of startup it happens. As you can see from the test, we have bugs on file for pretty much everything that isn't inevitable, so we're not singling out user.js unduly - it's just that, like all the other IO that we filed bugs for, it serves no point for the majority of users. Note that this is the nightly copy of the test, so some items will have been removed that you'd still see in strace logs of release. The additional Linux data is very helpful, though it will need some refining before it can be used. Really, we also need to add stat support to the linux file hooks so that we can get this data into that automated test (so we find out if more IO is added).

it instead ultimately exits with SEGFAULT: (sounds like that needs to be fixed) [1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594 [1767, Main Thread] ###!!! ABORT: file resource://gre/modules/TelemetryEnvironment.jsm, line 594

This is shutdown timing out while waiting for the add-on manager. It looks like it's tracked in bug 1513855.