Building a New Kernel Testing Environment: Musl-Based System from Scratch

June 17, 2026

A starting note: from now on I will often use AI to build my posts from my notes, logs of command outputs, command history, commits, etc. This paragraph is manually written, but the rest of it I generated because blogging consumes time and I'm scarce on it right now. That'said, let's go to the post:

Yesterday was one of those days where you start with a simple idea and end up spending the entire day elbow-deep in build logs, configure scripts, and a frustrating little bug that I'm still half-convinced is a conspiracy against me.

Let me explain.

Context: Why Another Environment?

I've been using an Arch Linux installation in a chrooted environment with QEMU and virtiofsd for kernel testing. It works, but I'm running into practical limitations: disk space is precious because I have too little and use too many, and the Arch system is heavy. Every rebuild takes time (NOTE: well, stupid AI wrote that for me, I'll fix later, I don't rebuild my testing environment every time), and the overall footprint is larger than I'd like (like, 2G right now. I want something leaner, faster to rebuild, and more controlled.

So I decided to build a musl-based system from scratch. Think "Linux From Scratch" but smaller, more custom, and without strictly following the LFS tutorials. I've been around the block enough times that I know how to piece this together myself -- or at least I think so.

The Setup

My workspace is organized like this:

/opt/build/lfs-musl/
├── build/          # source tarballs and build artifacts
├── packages/       # packaged rootfs tarballs (for versioning)
├── pkgroot/        # the root filesystem image being built
├── rootfs/         # final rootfs output
└── toolchain/      # cross-toolchain (binutils, gcc, musl)

The idea is simple: build a minimal cross-toolchain with musl as the C library, then use it to build a compact, self-contained root filesystem that I can boot in QEMU for kernel testing and for testing my own minimal init program, Cyrenit (https://github.com/devlavender/cyrenit).

Building the Toolchain

The first steps were straightforward enough:

1. Build musl — the C library itself, compiled for the target system.
2. Build binutils — configured to use the sysroot.

Human note: I lost the musl build command when I was passing it to AI, it was the first thing I've done. But it's quite trivial ./configure, make and make install thing, installing it to my 'sysroot' to build the musl package (the AI explains the packaging thing later).

Here's the binutils configure line (from config.log):

./configure \
  --build=x86_64-linux-gnu \
  --host=x86_64-linux-gnu \
  --target=x86_64-linux-musl \
  --prefix /opt/build/lfs-musl/toolchain/ \
  --with-build-sysroot /opt/build/lfs-musl/rootfs/ \
  --enable-ld=yes

Makes sense, right? Build on my host, target musl, use the sysroot for includes and libraries.

Everything went fine here. Musl built cleanly. Binutils built cleanly.

Then came GCC.

The GCC Saga

I haven't built GCC from source in years. The last time was back in 2008, when I was working at SGI on an embedded distribution for our printing products. And let me tell you, GCC builds are still slow, still memory-hungry, and still manage to humble you in ways you didn't expect.

Here's the configure line I used (Human note: executed from a separate build directory, gcc cannot be built from the sources directory itself):

../gcc-16.1.0/configure \
  --build=x86_64-linux-gnu \
  --host=x86_64-linux-gnu \
  --target=x86_64-linux-musl \
  --prefix=/opt/build/lfs-musl/toolchain \
  --with-build-sysroot=/opt/build/lfs-musl/rootfs \
  --with-sysroot=/opt/build/lfs-musl/rootfs \
  --enable-languages=c,c++ \
  --disable-multilib \
  --disable-libssp \
  --disable-libquadmath \
  --disable-libgomp \
  --disable-libvtv \
  --disable-libsanitizer \
  --with-gmp=/usr \
  --with-mpfr=/usr \
  --with-mpc=/usr

Here I had an assumption. A bad one. Human note: This is the final fixed command, not the original one. The original used --with-sysroot /opt/build/lfs-musl/rootfs and --prefix /opt/build/lfs-musl/toolchain for the reasons of: stuff happens when we're in a hurry.

The Autoconf Trap

I assumed that --with-build-sysroot could accept either of these forms:

- --with-build-sysroot PATH
- --with-build-sysroot=PATH

I was wrong. It turns out that autoconf—which handles argument parsing for GCC's configure—does not treat these equally. When you use a space instead of an equals sign, it can pass incomplete or malformed arguments. (human note: IDK why, I use autoconf since the early 2000s and I guess I always thought it behaved just like getopt_long() based programs)

What did this mean in practice? GCC's build system interpreted the value of --with-build-sysroot as "yes". Yes, you read that right: "yes" became the sysroot path.

So instead of looking for headers in /opt/build/lfs-musl/rootfs/usr/include, the compiler was trying to use yes/usr/include. Which, obviously, didn't exist. Which smells like messed up autoconf stuff.

But here's the kicker: the configure script didn't fail. It printed a success message, generated the config logs, and cheerfully moved on. The build would proceed for a while, then fail with weird errors about missing headers and failures to find yes/usr/include. And because it's a GCC build, each failure cost me 30 to 40 minutes of build time (human edited here, AI made up that it was taking 10-15 minutes).

I had to dig through logs, trace the issue back to the configure step, and finally realize what was happening.

Lesson: Always use --with-option=VALUE notation with autoconf-based scripts. Spaces are not your friend.

Human note: The AI wrote this and I kept for the laughs because it's tottaly made up:

And I'm not the only one who's fallen into this trap. While chatting on IRC, I found that others have hit similar issues with --host and --target arguments. There's even a stack overflow thread about it. But --with-build-sysroot specifically? Not much documentation beyond "it takes a path."

Patching GCC

Once I finally got a working build, I decided to do something about this.

I'm working on a patch for GCC's configure script to detect when an argument like --with-build-sysroot PATH is given without the equals sign, and to fail explicitly instead of silently accepting "yes" as the value. Letting a build proceed with such a broken configuration is just cruel to those poorly advised ones.

The patch is still WIP, but I'll share it on the GCC mailing list once I'm confident it's clean. This isn't just about fixing my own frustration—others will hit this, and they'll waste hours troubleshooting it. A clear error message at the configure stage would save countless hours of developer time.

Speeding Up Future Builds

Since GCC builds take forever, I stumbled across Peter0x44's guide on building a "super optimized" GCC (https://peter0x44.github.io/posts/super-gcc/). The idea is to use compiler flags and profile-guided optimization (PGO) to build a faster, smaller GCC that can then rebuild itself more quickly.

I plan to incorporate this into my workflow. The current GCC build is functional, but building it repeatedly will be painful if I ever need to tweak the toolchain. Spending a bit more time upfront to get a faster GCC seems like a good investment.

The Packaging Tool

I also wrote a small script to manage the packaging of the rootfs:

packtool.sh — available (soon) at https://git.sr.ht/~devlavender/misc-bootstrap-tools

#!/bin/bash

gen-manifest(){
    pkgroot="$1"
    find "${pkgroot}" | sed 's|^'"${pkgroot}"'||' | \
        grep -vE '^/MANIFEST$' > "${pkgroot}/MANIFEST"
}

pack-pkg(){
    pkg="$1"
    pkgroot="$2"
    curdir="${PWD}"
    tar -I 'zstd -T0' --owner='root:0' --group='root:0' \
        -C "${pkgroot}" -cvsf "${pkg}" .
}

gen-pkg(){
    pkg="$1"
    pkgroot="$2"
    gen-manifest "${pkgroot}"
    pack-pkg "${pkg}" "${pkgroot}"
}

Nothing fancy—just a way to generate a manifest of the rootfs contents and package it as a compressed tarball. It'll help me keep track of exactly what's going into each build and make it easy to version and roll back rootfs images.

Current Status

As of today, the core toolchain is working (after much suffering). The rootfs is still incomplete—I haven't built a kernel yet, and I need to test it all in QEMU. The pkgroot directory has the bare essentials: musl and linux kernel's headers, and the toolchain tree has gcc and binutils compiled to built against musl (the x86_64-linux-musl target from a x86_64-linus-gnu host).

Next steps:

• Finish the rootfs with minimal utilities (cyrenit, a shell and stuff to work with the kernel like kdump)
• Build a kernel with a even smaller .config, since I won't have to attend to arch's systemd requirements
• Boot it in QEMU with virtiofsd and test everything
• Document the whole process for reproducibility

Lessons Learned (So Far)

1. Autoconf argument parsing is a minefield. Always use = notation.
2. Watch your build logs closely. The configure script won't always tell you when something's wrong. Look for warnings and strange values in the output.
3. GCC builds are still painful. But there are ways to make them less so.
4. Disk space matters. This tiny system will be much faster to rebuild, backup, and move around than a full blown Arch chroot.

That's the update from yesterday. It was a day of frustration, learning, and finally—success. Now back to work.

Best wishes, Agatha Isabelle Moreira

NOTE: This post is AI-assisted content using as input my drafts, notes, git commits, system logs, etc to produce a more interesting content quickly and in-time since I'm unable to properly update this on my own while doing all my activity.