Making it easier for F-Droid to package Mozilla Firefox - Apps - F-Droid Forum

@Manizuca,

Manizuca https://forum.f-droid.org/u/manizuca
December 1

nalexander:

Can you explain what this means? Do binaries mean things like proprietary
codec blobs? I didn’t see anything like this when I looked in the F-Droid
build scripts at https://gitlab.com/fdroid/fdroiddata/blob/master/ 1
metadata/org.mozilla.fennec_fdroid.txt#L3959-3989.

Yeah, the build script calls a script
https://gitlab.com/Manizuca/fenneclocales/raw/master/prebuild.sh 6 within
the same repo that holds the locales. These files are removed because the
fdroid-server checks for the presence of any executable/binary data.

Oh my. In general, this approach is always going to be broken, and Mozilla
would much prefer to hear about and potentially accommodate your use cases
than have you require such blunt instruments. But this approach does have
the advantage of not requiring upstream changes, which I understand!

nalexander:

More details would be helpful here. I would expect that --disable-tests
in your mozconfig wouldn’t fail the builds, but I don’t know what’s at
play.

If i remove just the files, but keep the .ini s, i get something like

0:25.26 ==============================
0:25.26 ERROR PROCESSING MOZBUILD FILE
0:25.26 ==============================
0:25.26 The error occurred while processing the following file or one of
the files it includes:
0:25.26 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:25.26
0:25.26 The error occurred when validating the result of the execution.
The reported error is:
0:25.26 Error processing test manifest file /Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini: Traceback
(most recent call last):
0:25.26
0:25.26 File “/Sources/F-droid/build/org.fdroid.fennec/python/mozbuild/mozbuild/frontend/emitter.py”,
line 1294, in _process_test_manifest
0:25.26 path, ', '.join(missing)), context)
0:25.26
0:25.26 SandboxValidationError:
0:25.26 ==============================
0:25.26 ERROR PROCESSING MOZBUILD FILE
0:25.26 ==============================
0:25.26 The error occurred while processing the following file or one of
the files it includes:
0:25.26 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:25.26
0:25.26 The error occurred when validating the result of the execution.
The reported error is:
0:25.26 Test manifest (/Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini) lists test
that does not exist: test_asyncadd.js, test_asyncbadadd.js,
test_asyncbadremove.js, test_asyncremove.js, test_bug399727.js,
test_bug419769_1.js, test_bug419769_2.js, test_bug425768.js,
test_bug433248.js, test_bug446708.js, test_bug467740.js,
test_createempty.js, test_deflatedata.js, test_directory.js,
test_editexisting.js, test_storedata.js, test_sync.js, test_undochange.js,
test_zipcomment.js, test_zippermissions.js, test_bug717061.js,
test_alignment.js

And if i remove all the directory with the conflicting data including the
.ini s, the error is:

0:23.25 ==============================
0:23.25 ERROR PROCESSING MOZBUILD FILE
0:23.25 ==============================
0:23.25 The error occurred while processing the following file:
0:23.25 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:23.25
0:23.25 The error was triggered on line 7 of this file:
0:23.25 XPCSHELL_TESTS_MANIFESTS += [‘test/unit/xpcshell.ini’]
0:23.25
0:23.25 An error was encountered as part of executing the file itself. The
error appears to be the fault of the script.
0:23.25
0:23.25 The error as reported by Python is:
0:23.25 [‘IOError: Missing files: /Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini\n’]

In this example happens with a xpcshell.ini file, but the same happens
with something like MOCHITEST_MANIFESTS += [‘test/mochitest.ini’]. I
opted to remove all the test directories with conflicting files, and then
grep the moz.build to remove the .ini s

Why is it that you need to do anything with these test files? They don’t
feature in the packaged artifacts at all, and if you use --disable-tests
in your mozconfig you should not really see any interaction with them.

nalexander:

These should all be supported as mozconfig options. If they’re not doing
the right things for you, I consider that a bug, and would appreciate being
CCed on a ticket in the Core :: Build Config component at
https://bugzilla.mozilla.org 1. In particular, Telemetry is growing in
scope and intrusiveness, so I wouldn’t be surprised if we need a little
focused attention to ensure we’re not accidentally pushing bits to Mozilla
that F-Droid users don’t want exposed.

There was some problems with telemetry code being included even when
disabling it with .mozconfig. I see that many of them were fixed in the
last versions.

nalexander:

Firefox for Android has been pretty focused on using newer technologies to
A/B test and really understand our user base, so we’ve used Adjust (a
German tracker with a good privacy policy), Switchboard (an A/B module that
we run an in-house backend for), and Leanplum (a marketing application
about which I know little). It’s my belief that we’ll be able to make
building with Google Play Services imply no Adjust and Leanplum. I don’t
know what the right outcome for Switchboard is for F-Droid – you don’t want
Mozilla’s instance, but you probably don’t want to run your own (which has
a non-trivial cost). I’m pretty sure that everything will work if you “opt
out” of Switchboard, but I don’t know if that’s easy to arrange at
build-time. We can try to make it so.

Definitely i have to test the last versions, but with a packet analyzer we
discovered some time ago that some information was being sent even when
opting out from the user preferences. As i said, the technology used by
firefox for android has changed a lot since then.

I would appreciate you revisiting, if and when you can, but I don’t think
this should be high priority for you.

nalexander:

Surely the majority of F-Droid’s packages are built with Gradle, so this
problem must be addressed in some way. Firefox for Android’s Gradle
configuration is pretty complex, but it should “just work” to run mach
build after the dependencies are in place. I expect that F-Droid could
work through these dependency issues fairly easily. I can provide links to
what Firefox for Android does: the tl;dr is that we collect Gradle
dependencies using a proxy Maven repository running Nexus Sonatype. It’s
not that complicated; start reading at https://searchfox.org/mozilla- 2
central/source/taskcluster/scripts/misc/android-gradle-dependencies.sh
and the linked scripts.

I will try a gradle build with more recent versions (last time i tried was
Fx54), but if you use a custom maven repo it could be problems as there are
just a couple of them whitelisted. @hans
https://forum.f-droid.org/u/hans should know more.

Mozilla does not run its own Maven repository, and is unlikely to depend on
one anytime soon: it’s a non-trivial cost when you run continuous
integration at the scale that we do. The set of Maven dependencies evolves
slowly, so when it changes we capture the entire dependency set, archive
it, and offer that to subsequent builds. Right now, those dependencies are
fetched from jcenter (run by Bintray) and maven.google.com 2 (run by
Google). It might be possible to avoid Google’s repository right now, but
the evolution is definitely towards everything Android related being from
there so it’s probably not possible to avoid for long.

In any case, I don’t think you would have big problems doing what we do;
and, if you so chose, you could also use Mozilla’s dependency archives –
they’re just .tar.xz files of Maven file repositories.

Firefox 58 will probably be the last Firefox for Android that will not
require Gradle to build – watch
https://bugzilla.mozilla.org/show_bug.cgi?id=1414415 5
to see when the non-Gradle build is removed. In order to have a smooth
transition as Firefox 58 becomes release, F-Droid will need to do some work
to support this.

nalexander:

Thanks for your feedback!
Nick

Many thanks to you! [image: ]

Is it easy to test changes to F-Droid’s build script? I think we might be
able to do a little focused work and get a much simpler script that
achieves the same outcomes.

Best,
Nick