Cutting pkgsrc-2021Q3

The BSD model is roughly the same as the traditional Unix model of what constitutes the operating system. Kernel, core programs, core libraries, and a toolchain to make new programs. In NetBSD’s case, it also includes a monolithic X Window System tree (carefully nested underneath the /usr/X11R7 hierarchy).

It means that you get a hopefully stable core OS upon which to build stacks of third-party software, all of which gets installed to an isolated hierarchy so it doesn’t touch the core OS. Or an unstable core OS to build on, if you like - whichever’s your thing.

pkgsrc is an auxiliary package manager. It assumes your OS includes all the basic Unix utilities and a toolchain, and uses those to provide packages to supplement whatever your OS provides.

While the NetBSD 9 stable branch (released early 2020) is probably going to be in-support for a few more years, NetBSD 8 (released 2018) is certainly on the way out. Notably, the version of OpenSSL used in NetBSD 8 is now very difficult to keep in sync with upstream OpenSSL without breaking compatibility.

The overal model of NetBSD means that the system compiler is fixed at gcc version 7.4.0 in NetBSD 9, and gcc 5.4.0 in NetBSD 8.

GCC 7.4.0 includes initial support for C++17, which is becoming increasingly mainstream. However, some C++17 features, which are beginning to be seen in the wild, are only available in GCC 8.

The notion that the compiler is stable is an idea that comes into conflict with software developed primarily for Linux - if you have an old compiler, surely you want an old release of the software too, and you’re running Debian Stable or something. Building newer software releases with the system compiler becomes increasingly difficult as the OS release ages, even if it’s only aged by a year.

pkgsrc leaves open the opportunity to require a newer compiler for specific packages. However, this may introduce problems when linking dependencies that have been built with a mixture of older and newer compilers.

Another option is to require a newer compiler to be bootstrapped for use with pkgsrc. This is contentious, as it’s generally accepted that things should Just Work with NetBSD stable branches, and integrate well.

Other system libraries also pose problems, for example, the X11 libraries shipped with NetBSD 9 do not meet the freshness requirements imposed by GNOME 41.

Every 3 months, pkgsrc goes into a (usually week long) commit freeze. No normal package updates, only fixes, and security updates. The hope is that enough testing is performed on enough targets to iron out any major issues in preparation for cutting a stable quarterly branch which will last for the next 3 months. Hopefully, developers' focus is also shifted a bit.

This stable branch is used as the basis for NetBSD binary package builds on at least 10 CPU architectures.

The Freeze is a contentious issue. Often, pkgsrc sees a sudden rush of commits as soon as a upcoming freeze is announced, and a sudden rush of commits as soon as it ends. These rushes often introduce surprise breakage. Maybe they should be surprising? :)

Sometimes freezes include emergency rollbacks. In recent pkgsrc history, Samba has been rolled back during a freeze - an update introduced a non-trivial runtime error on BSD platforms.

A recent update to Rust 1.54.0 just before freeze introduced a symbol conflict on 64-bit ARM that resulted in problems with librsvg (a very popular library used by many pieces of software to support SVG images). Hopefully, a quick solution is found that doesn’t involve a rollback.

Recently I obtained access to a new dedicated server for performing large-scale builds of pkgsrc - 64GB RAM, 8 AMD Ryzen 7 3700X cores, 16 threads, NetBSD 9.99.x.

My bulk building setup involves creating 16 chroot sandboxes which are used to perform parallel pkgsrc bulk builds on 16 threads for amd64 and i386, on NetBSD -8, -9, and -current. Using chroots and null mounts ensures each builder has an isolated view of the system (or an isolated view of a slightly foreign system, be it an older OS release or a 32-bit one), and allows for low overhead communication across the builders.

A full build of pkgsrc for a single target, roughly 24400 packages currently, takes a day or two on that hardware. It’s important to get quick feedback on whether the whole thing is working, as well as to provide quick updates of working binaries to users.

As a surprise to users, sometimes some experimental options will be enabled in my builds too!

pkgsrc-2021Q3 will include in its release notes:

After the freeze ends, there is ample time to introduce changes for people to try as the new branch is developed. After pkgsrc-2021Q3, I’m hoping to enable some more useful security hardening options by default. There is also the obligatory switch to using new major releases of programming languages like Python by default in non-version-prefixed packages.