Upgrade StorageDistros building Zotero from source
Hey, I successfully built Zotero from source in nixpkgs (NixOS) [1]. The creation of this build recipe was rather time intensive and painful. I hope you get an idea on the requirements of distributions to make it easier for us in the future :).
When starting, I learned that Alpine Linux is the only other distro building Zotero (and as part of it, Firefox) from source [2]. Its difficulties are (for Zotero 7.0.30) to build Firefox 115 with more recent dependencies (like cbindgen). Additionally, it has to patch some stuff to run with musl instead of glibc.
With nixpkgs, the difficulties are very different: During build, no network access is possible. This also means you cannot "just" do `npm install` wherever you like. Instead, all inputs have to be explicitly downloaded and being immutable (checked via hash). Helper functions make it easy to e.g. build NPM packages.
Some concrete suggestions (sorted from small/easy to complex, not sure how actionable they are):
- Add the icons/ directory during build [3]
- Don't try to download en-US locales during pdf-reader and note-editor builds (patch deactivatin the download [4], and instead copying them from the main zotero library in [5])
- You always expect the zotero git submodules to be in their designated subfolders, and you can cd into them and run e.g. `npm ci` or `npm run build`. This is problematic for packaging for multiple reasons. It would be better to instruct npm (not sure how, not a js developer myself) to require the submodules as npm dependencies. This way, you can access and depend on the finished build assets below node_modules/. This would allow us to separately build the submodules (already the case) and just provide their finished build instead of constantly copying the whole build directory into dependents (new), e.g. at [6]. I think you have to create a npm `install` script for this that builds the project and selects files/folders to include in the "build output".
- Offer a way to supply a pre-existing Firefox package directory instead of always downloading it (to avoid this lengthy patch [7])
- Finally, requiring the .git folder to get git hashes is somewhat difficult for nixpkgs, so I patched the dependency out [8]. While I understand why you retrieve the main git hash to report it as part of the version, the hashes for the submodules during build might be avoidable.
- Also have a look at the build script and patches of Alpine Linux.
This might lead other distros to build Zotero from source as well sometime down the road.
Feel free to reach out if you want to implement some of these things.
Best, Mynacol
P.S.: I appreciate your plan to update Zotero more frequently in the future. This hopefully means that you will switch reasonably fast from one Firefox ESR version to the next before they run out of support and build scripts are dropped.
[1] https://github.com/NixOS/nixpkgs/pull/483099
[2] https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/zotero
[3] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/build-fixes.patch#L33
[4] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/pdf-reader-locales.patch
[5] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/package.nix#L101-L102
[6] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/package.nix#L53-L60
[7] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/avoid-xulrunner-fetch.patch#L25-L135
[8] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/avoid-git.patch
When starting, I learned that Alpine Linux is the only other distro building Zotero (and as part of it, Firefox) from source [2]. Its difficulties are (for Zotero 7.0.30) to build Firefox 115 with more recent dependencies (like cbindgen). Additionally, it has to patch some stuff to run with musl instead of glibc.
With nixpkgs, the difficulties are very different: During build, no network access is possible. This also means you cannot "just" do `npm install` wherever you like. Instead, all inputs have to be explicitly downloaded and being immutable (checked via hash). Helper functions make it easy to e.g. build NPM packages.
Some concrete suggestions (sorted from small/easy to complex, not sure how actionable they are):
- Add the icons/ directory during build [3]
- Don't try to download en-US locales during pdf-reader and note-editor builds (patch deactivatin the download [4], and instead copying them from the main zotero library in [5])
- You always expect the zotero git submodules to be in their designated subfolders, and you can cd into them and run e.g. `npm ci` or `npm run build`. This is problematic for packaging for multiple reasons. It would be better to instruct npm (not sure how, not a js developer myself) to require the submodules as npm dependencies. This way, you can access and depend on the finished build assets below node_modules/. This would allow us to separately build the submodules (already the case) and just provide their finished build instead of constantly copying the whole build directory into dependents (new), e.g. at [6]. I think you have to create a npm `install` script for this that builds the project and selects files/folders to include in the "build output".
- Offer a way to supply a pre-existing Firefox package directory instead of always downloading it (to avoid this lengthy patch [7])
- Finally, requiring the .git folder to get git hashes is somewhat difficult for nixpkgs, so I patched the dependency out [8]. While I understand why you retrieve the main git hash to report it as part of the version, the hashes for the submodules during build might be avoidable.
- Also have a look at the build script and patches of Alpine Linux.
This might lead other distros to build Zotero from source as well sometime down the road.
Feel free to reach out if you want to implement some of these things.
Best, Mynacol
P.S.: I appreciate your plan to update Zotero more frequently in the future. This hopefully means that you will switch reasonably fast from one Firefox ESR version to the next before they run out of support and build scripts are dropped.
[1] https://github.com/NixOS/nixpkgs/pull/483099
[2] https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/zotero
[3] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/build-fixes.patch#L33
[4] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/pdf-reader-locales.patch
[5] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/package.nix#L101-L102
[6] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/package.nix#L53-L60
[7] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/avoid-xulrunner-fetch.patch#L25-L135
[8] https://github.com/Mynacol/nixpkgs/blob/zotero-from-source/pkgs/by-name/zo/zotero/avoid-git.patch