2

Ask HN: Is anyone successfully self-hosting Firefox Sync?

 2 years ago
source link: https://news.ycombinator.com/item?id=34674569
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Ask HN: Is anyone successfully self-hosting Firefox Sync?

Ask HN: Is anyone successfully self-hosting Firefox Sync?
202 points by hosteur 6 hours ago | hide | past | favorite | 66 comments
I use Firefox as my primary browser, and I would absolutely love to be able to use the Firefox Sync functionality to sync bookmarks, etc. across several devices.

However, I do not want to risk storing absolutely sensitive data (browsing history, auth cookies, credentials, etc) at some third party.

Everytime this comes up in HN comments, someone says that Firefox Sync can be self-hosted. And while this was true in the older versions of Firefox, it seems to me that self-hosting is currently neither supported nor documented. And it has been like this for at least since 2020[1][2]. It seems self-hosting is not a priority for Mozilla.

Is anyone actually self-hosting Firefox Sync? If so, how do you do it? How is it working for you? Any issues to be aware of?

[1]: https://github.com/mozilla-services/syncstorage-rs/issues/681 (opened on Jun 22, 2020)

[2]: https://github.com/mozilla-services/syncstorage-rs/issues/550 (opened on May 26, 2020)

Yes, I'm running it with the mozilla/syncserver docker image[1], but still using the hosted identity service. The setup procedure is a bit fickle, the error handling on Firefox' side is quite bad, sometimes the first sync times out when you have a larger data set, and you need to manually enable each sync type to reduce the size. But once it's up and running, I don't really have any issues.

I do hope that syncstorage-rs will become usable eventually and hopefully improve performance.

Note: contrary to your post, cookies are not synced.

[1]: https://hub.docker.com/r/mozilla/syncserver

I've self-hosted the original Firefox syncserver[0] for a couple of years now. It's a now unmaintained Python 2 + gunicorn web app.

It still uses the Firefox identities service, since I haven't bothered to try setting that one up myself. That means I need a Firefox/Mozilla account even with a self-hosted sync server, which isn't ideal.

Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted.

I originally went through the setup instructions manually. Now I've created a Nix package[2] and NixOS module for it[3], which was an adventure by itself. I can now easily reproduce the setup, which I did while migrating my webserver to a different provider.

As for why I didn't try syncstorage-rs, it's rather petty, but the old syncserver was working well for me, and it doesn't yet support SQLite.

[0] https://github.com/mozilla-services/syncserver

[1] https://github.com/mozilla-services/syncserver#removing-mozi...

[2] https://git.sr.ht/~williamvds/config/tree/master/item/nix/pk...

[3] https://git.sr.ht/~williamvds/config/tree/master/item/nix/mo...

s.gif
The first few lines in the repo you mention illustrate the problem:

> Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered.

And its replacement syncserver-rs is in this forever-unfinished state so they've effectively killed self-hosting. At least from a support standpoint.

s.gif
Good point, I'll update the comment to indicate syncserver is now unmaintained.

Of course, everything self-hosted is at my own risk etc. And official support is a luxury in this kind of FOSS, I generally don't count on it.

But since it works for me, it may be worth sharing the details.

s.gif
I agree that a lack of support isn't a huge issue for this kind of thing. I'm more worried about the maintenance, like a vulnerability remaining unpatched (it being python 2 doesn't help either) or a protocol change that makes the old server suddenly unviable.

I really wish they'd just finish the Rust version. That would solve everything.

s.gif
Have you considered upstreaming the package and module to nixpkgs? It would be awesome to have those one line of config away for all nixpkgs users.
s.gif
I briefly considered it, but there are some big problems:

- The new replacement package, syncserver-rs, has already been added to Nixpkgs, along with a NixOS module [0]

- Python 2 and the package itself are no longer supported, meaning...

- support in Nixpkgs is pretty rough - dependencies have been updated such that they no longer support Python 2, or have been removed outright, meaning...

- syncserver has already been removed from Nixpkgs

- in order to avoid the problem of unsupported dependencies, I delegated dependency resolution to a community project, poetry2nix[1]. Even so, it was a nightmare to get working, lots of hacks and workarounds were needed

Of course, my repo is public, so anyone is welcome to use the Flake within. If they do, I'd suggest dropping me a line so I don't do anything drastic to the package or module.

[0] https://github.com/NixOS/nixpkgs/pull/176835

[1] https://github.com/nix-community/poetry2nix

s.gif
Yeah, Python2 alone is enough to not bother. Hopefully the new version "ships" soon.
Shameless plug: A while ago I wrote a cli for firefox sync [1] (in my case mostly to automate some stuff when I create bookmarks).

It does not really solve your problem for browser-history/credentials etc. But as long as you use a client that you verified you can be kinda sure the data is E2E encrypted and can't be read by Mozilla.

Also on a side note: To this day the weave protocol is one of the more (needlessly) complex protocols I ever implemented and I'm still not sure what problems some of the design decisions solve...

[1] github.com/mikescher/firefox-sync-client

> However, I do not want to risk storing absolutely sensitive data (browsing history, auth cookies, credentials, etc) at some third party.

FWIW Firefox says they do not have the capability to read your data: https://hacks.mozilla.org/2018/11/firefox-sync-privacy/

s.gif
Unfortunately, this isn’t really true. The party which holds your data is also the party which ships evergreen versions of the software used to decrypt that data. There’s nothing to stop the government in Firefox’s jurisdiction from bullying them into adding a backdoor. We can assume the bullying is ongoing.

I like to imagine it as having a safe deposit box provided by a local bank. I have the only key and have proven that to be the case. I also store the key at home in another safe which is regularly serviced by bank employees. While they probably don’t abuse their position in the latter to access the former, it would be completely wrong to say they do not have the capability to get at my stuff.

All that aside, I love FF sync and rely on it everyday for work stuff. It is like magic and I’m very grateful for the feature. I am realistic about the NSA’s level of interest in my work calendar and the threat they pose to my day to day life.

s.gif
> The party which holds your data is also the party which ships evergreen versions of the software used to decrypt that data. There’s nothing to stop the government in Firefox’s jurisdiction from bullying them into adding a backdoor.

Hosting your own sync server doesn't prevent this.

s.gif
The government can’t bully you into decrypting your own data or releasing your encrypted data without you knowing.
s.gif
If Firefox is backdoored, as per GP's hypothetical situation, your data can be sent to a sync server other than the one you configured.

The only realistic way to stop it would be to only use a specific audited version of FF. If that's the case then there's no need to use your own sync server because you already know it's not backdoored.

s.gif
Which government? Because there absolutely are governments that can, UK for example.
s.gif
They can bully the open sourcerer into adding a backdoor to the code, then use that backdoor without you noticing.
s.gif
Banks always drill out the lock on a safe deposit box if they need to get in, even though it wouldn't be hard for someone to pick the lock (or so lockpicks who claim to have the ability to check this say, I've never tried to pick the lock on a safe deposit box, and my lockpick skills are not very good so I doubt I could)
s.gif
Then you can’t trust the FF code not to send a copy to their own servers also.
s.gif
Why would you need to trust it? Just block access to their servers on your network.

Anyone _that_ concerned can operate a whitelist-only policy on their network, now nothing goes anywhere they don't want it to.

s.gif
White-listing what addresses a web browser can reach seems to go against the intended use - you know, to browse the web.

Edit: Also, I don't think this is as much about trust as it is about avoiding being part of the huge target that the centrally hosted Firefox Sync servers must be.

While Firefox developers and Mozilla might be wholly trustworthy, they might become compromised by some configuration error or zero day vulnerability. So someone might steal the sweet sweet treasure that is in the Sync servers. Now, that is encrypted of course, but they might have made a simple error[1] in how encryption is implemented and the encryption may be defeated.

[1]: See for example: https://www.schneier.com/blog/archives/2008/05/random_number...

s.gif
If you have something you are that concerned about then you should whitelist everything. Note that the whitelist needs to be one both sides, the server and the users. This along with lots of other security protections, some of which are a lot stronger than a whitelist.
s.gif
> I have the only key and have proven that to be the case

Locksmiths will get through that lock in a couple of minutes, probably without you ever knowing

The only thing stopping the bank and/or government from accessing your box is the law

s.gif
This is what I learned over the years after my techno fantasies were confronted with the real world. Technology can give you good protections to make blanket or mass interference on people’s affairs costly, but the illusion of safety it gives is over emphasised.

A very motivated actor will just knock on your door and ask you for the key(digital or otherwise) at gun point and it is game over.

Like one of the replies above, model your threat and enjoy like free of paranoia. It is a beautiful world out there! Ah and be politically active so your representatives preserve common freedom and well being.

s.gif
And we all know from Snowden/Assange how effective that is.
s.gif
> FWIW Firefox says they do not have the capability to read your data: https://hacks.mozilla.org/2018/11/firefox-sync-privacy/

The article is from 2018. Isn't this only relating to the old version of Firefox Sync - the one in which self-hosting is actually supported?

The design looks reasonable. However, my main problem with it is that putting all this extremely valuable data in one place makes it a huge target. A supply-chain attack or an implementation error in the crypto would be devastating.

s.gif
That's also an option in Chrome, "encrypt sync data with your own sync passphrase"
I faced enough issues that I switched to the Mozilla endpoints. Problems ranged from no clear documentation for configuration, unclear resource usage, no documented method to host identity as well. My one hope was to get my data exported via SQLite for fun stuff, as an automated export. But the data format was also not well documented, so that never happened.
Why does every "Ask HN" I see come with a grey text color? Is this a stylistic thing or do people take issue with people asking questions and downvote?
s.gif
HN does the same when you go through your settings and select "show hidden"... You "see" them but you dont.. The light gray goes even closer to the background color in this case.
s.gif
HN does this purposefully to show its preference for external content rather than self posts.
s.gif
I wonder if there could be an alternative way of getting the point across, as opposed to having low contrast text, which some tools report as an accessibility problem: https://wave.webaim.org/report#/https://news.ycombinator.com...

But other than that, it's an interesting design choice.

s.gif
HN does this because they don’t care. It wouldn’t take much to make this site usable, but they refuse to add 5 lines of CSS to make the site work in dark mode.
s.gif
Doesn't sound like a good default. I used those before, but I shouldn't have to to make a website barely usable.
s.gif
Aaaaah facepalm. Using this for a decade, never had a clue.
s.gif
If something is bothering you, it seems a bit weird to allow it to keep bothering you when the solution is so trivial.
I found the lockwise app very useful for FF sync and have no idea why they shitcanned it.

I don't really understand how Mozilla continues to function at all as an org.

s.gif
The functionality is (mostly) built into Firefox now so there’s little need for a separate app. You can enable native app autofill for both Android and iOS. There are a few things that should still be done to improve the experience but it doesn’t seem like a priority now.
I was thinking about doing this with my own custom setup. Im thinking of putting the folder .mozilla into a gocryptfs encrypted folder and use syncthing to sync it between devices.
Yes, and I went the whole mile. I wrote a couple of comments on it previously[0][1], which at least lay out the required images for a full stack. As you can see it doesn't use the newer release but it still runs fine on latest FF. Though you probably want the new version. Can't imagine it being significantly worse to get up and running now.

TLDR: If you're OK with piggybacking on them for auth, it's straightforward, easy and lightweight. If you want to be fully self-sufficient and leak no data you need to either implement a replacement for fxa (seriously this would be appreciated and maybe less effort if you're coming in fresh) or host that whole machinery of microservices - this is where the head-scratching starts and you need some dedication for it.

I encourage other people to do it, and do go to whichever is more appropriate of the Matrix room or the GitHub Issues and engage when you hit things that are out of date, unclear, or broken.

[0]: https://news.ycombinator.com/item?id=30315816

[1]: https://news.ycombinator.com/item?id=30727935

What other self-hosted bookmark sync solutions exist that work with Firefox (and perhaps other browsers)?
s.gif
I used to use xbrowsersync [0] for cross-browser sync and rn I'm using raindrop [1]. Xbrowsersync is self-hostable, but raindrop is not. Both are FOSS.

There's also a big list of different projects here: https://github.com/awesome-selfhosted/awesome-selfhosted#boo...

[0] https://www.xbrowsersync.org/ [1] https://raindrop.io/

s.gif
ive been using floccus for a few months now and its been working ok so far.

it syncs to the webdav server on my synology but there's also the option to sync it using decsync (which is sort of like a local webdav server for each device) and some other sync service like syncthing or dropbox to sync the files.

floccus has an android app or it can also be installed as a normal extension with the kiwi browser.

i tried xbrowsersync first but i had some issue when i used it with vivaldi where a new copy of my bookmarks showed up in the "deleted bookmarks" section every time a sync happened, and i couldn't see any way of fixing that either which is why i went with floccus in the end

Anecdotally, I miss the days of Netscape combined with Netscape Enterprise Server. Ran it at college, and you could log in to any Netscape browser and all your bookmarks appeared. Then NES died, Netscape became Mozilla and it all went away.

I see there are bookmark sync extensions, but only bookmarks (and I didn't check if those extensions allow self hosting).

I suppose you could always run Firefox portable from a location that was Dropbox/Nextcloud/syncthing/whatever synced. Just don't run two instances at once, or the SQLite DBs will have a bad time?

s.gif
I miss Xmarks... I believe it was killed off completely when lastpass was aquired by Logmein, a couple of years ago. Xmarks really hit the spot for me for syncing bookmarks. It did only that, across browsers, across machines, and because It was limited to bookmarks sync, I had no qualms about using the same account on employer provided machines.

https://en.wikipedia.org/wiki/Xmarks_Sync

s.gif
> I suppose you could always run Firefox portable from a location that was Dropbox/Nextcloud/syncthing/whatever synced. Just don't run two instances at once, or the SQLite DBs will have a bad time?

I do something to that effect. However, it does not come without problems. Running parallel instances is one of them... Also, cross platform is a problem.

s.gif
Hypothetically, since I have no way to test this easily right now, you only need to run the profile directory in a synced location and run the exe from a platform-specific portable directory.
I self-host syncserver in Docker, it is relatively easy to set up. The main issue is that it would not tell you that it doesn't work, instead it would silently leave a log file in your browser (about:sync-log). Once it is up and running, however, it works flawlessly. You also have to consider that syncserver is unmaintained and uses Python 2, which has long been EOL.
I've got a related question but about more restricted capabilities:

I'd like to sync just bookmarks without joining the "Firefox ecosystem" or any other ecosystem.

Is there some bookmark sync extension with a self hosted sync solution or that will just sync via a (service similar to) Dropbox folder?

Edit: syncing via a git repo would be fine for example :)

Vaultwarden for Passwordsync and Wallabag for Bookmarks.

Con: no „real“ Firefox Sync - Pro: no „real“ Firefox Sync - it works on any device with any browser, data is self-hosted

s.gif
I set up Wallabag recently for this very purpose. My initial impression is that it mainly focuses on fetching and storing the content itself, rather than giving you an easy to navigate bookmark library. So it's great for, eg, saving long-form articles and blog posts, but not so good for bookmarking interactive/dynamic content.
s.gif
Just tried that, it's just broken (at least, those instructions).
i just copy the .mozilla folder over whenever i install a new device
s.gif
Do you not use bookmarks? Or are yours so stable it isn't any work to keep up?

This makes tons of sense to me with extensions and the other profile data, but not that.

unfortunately not anymore, in the early days of firefox sync back when it was called wave it was easy to selfhost. Just throw some[1] php files to a host and it worked. Then mozilla changed it to use multiple services (auth, ...) it got difficult and the documentation wasn't easy to access.

At that time i stoped selfhosting & using it :(

[1]: https://github.com/balu-/FSyncMS

s.gif
I think it's worth noting that that document links to the Github for the sync server, and that page says "Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered." So yes, you can probably get it running in docker, but bugs etc won't be fixed unless you're comfortable working on older Python code.
Firefox Sync uses send-to-end encryption.

So self hosting isn't as necessary.

No, I used to host the previous version but when that stopped working (including the workarounds which kept it going for a bit longer) I ceased using it. While the sync server itself is not that much of an effort to get going, the auth server is. I do not use external auth services when I can avoid it so I chose to forego on using sync instead of creating a "Firefox account". If I ever find an easier way to get the auth server running and if I still use Firefox by that time I might start using it again but until that time it is a no-go.

To Mozilla: fire your overpaid activist CEO Mitchell Baker and use the resulting funds to hire (back) the developers she got rid of, clean up the mess, get rid of the social justice nonsense and take back your place as a browser development organisation instead of a 'colours of change' peddler.

s.gif
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search:

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK