Build | GitHub | Mastodon | SourceHut |

This journal is part of series,

Previously, we try hard to make the "" [1] [2] script run the test and building our awwan[3] application, because we want a simple flow for our development, where executing "mkosi --directory=_mkosi --incremental" is enough. Unfortunately, it does not work.

In this attempt, I would like to take different approach. Instead of building and running the container in single "make", we booted up the container first and then trigger running the test from host using command "machinectl awwan@image …​".

The goals is

  1. to create an image, with user "awwan" created and sshd service enabled,

  2. to boot the image, and mount the awwan repository into "/home/awwan/src", and

  3. to test and build awwan application inside container using "machinectl shell …​" command

The container will be running until we stop it, which minimize disk read-write during development.

Rewriting the initial setup

Unlike previous journal, where we try and write output of each command, in this section I will dump each configurations and explain its content.

Start from the structure of our _mkosi directory,

├── mkosi.cache/
│   ├── gocache -> /home/ms/.cache/go-build
│   └── gomodcache -> /home/ms/go/pkg/mod
├── mkosi.conf
├── mkosi.conf.d/
│   └── archlinux.conf
├── mkosi.extra/
│   └── etc/
│       ├── ssh/
│       │   └── sshd_config
│       └── sudoers.d/
│           └── awwan
├── mkosi.finalize
├── mkosi.nspawn
└── mkosi.prepare


This directory contains the cache of downloaded system packages that we installed under "Packages=" option in "mkosi.conf".

Inside this directory, we have two symlinks created before we build the image: "gocache" and "gomodcache". Those symlinks are created using

$ ln -sf $(shell go env GOCACHE) _mkosi/mkosi.cache/gocache
$ ln -sf $(shell go env GOMODCACHE) _mkosi/mkosi.cache/gomodcache

This symlinks will be mounted into the container later when we boot the image, see "mkosi.nspawn" file below. It is used to cache the Go build and Go modules.

Without using symlink we will need full path, which is dynamic between users. For example, the Go build cache in my host would be "/home/ms/.cache/go-build". Using fixed path will not make our configuration usable to other developers.


The main configuration of mkosi. In this setup, this file only contains the common configuration, distro specific is moved to directory "mkosi.conf.d".




The "Format=directory" means the image output will be in form of directory structure, as we see later after building image completed.

The "Output=awwan-test" define the image name.

The "Bootable=no" means we are not creating image that is bootable using qemu. Does not mean that "mkosi boot" does not works, it just that we will not create boot partition and install kernel inside the image, so the image is lightweight container that is boot-able using systemd-nspawn[4].

From my understanding, the "CleanPackageMetadata=false" suppose to means that after the pacman databases downloaded (inside the OS tree), and then copied into the image for installing packages, it will not be removed once the image completed.

The "Incremental=yes" means that we enable "--incremental" build, if the image already build and no changes in the "mkosi.conf", "mkosi.prepare", or "mkosi.finalize"; running "mkosi boot" will not re-build the image again, but use the existing image as per "Output=".


This is the mkosi configuration specific to Arch Linux.



The "Distribution=arch" means that this configuration will be included only if distribution that we want to build is "Arch Linux".

The "SkeletonTrees=/var/lib/pacman/sync:/var/lib/pacman/sync" means that the content of host directory "/var/lib/pacman/sync" will be copied into OS tree at the same directory path. We use this to minimize database sync when running pacman.

The "Packages=" option list all packages to be installed into the image.


This directory contains files that will be copied after all packages installed. In this directory, we have two files. One is "etc/ssh/sshd_config" to changes the SSHD server to run on port 10022 instead of 22. Another one is "etc/sudoers.d/awwan" which contains sudo configuration for user "awwan" and "awwanssh".


This is a shell script that will be run by mkosi inside chroot to enable sshd service.


if [ "$container" != "mkosi" ]; then
	exec mkosi-chroot "$CHROOT_SCRIPT" "$@"

systemctl enable sshd.service


This is a template file for generating ".nspawn"[5] file after image completed.


In this file, when "systemd-nspawn" executed to run the image, it will mount host directory "../" (the awwan repository) into container "/home/awwan/src", "mkosi.cache/gocache" into container "/home/awwan/.cache/go-build", and "mkosi.cache/gomodcache" into container "/home/awwan/go/pkg/mod".


This is shell script that will be run by mkosi once after all packages are installed.


echo "--- mkosi.prepare: args=$@"
echo "--- mkosi.prepare: container=$container"

if [ "$container" != "mkosi" ]; then
	exec mkosi-chroot "$CHROOT_SCRIPT" "$@"

if [ "$1" == "final" ]; then
	set -x
	## User testing sudo with password prompt.
	## The UID of user in container must equal with UID in host, for
	## better compatibility.
	## The password is "awwan".
	useradd --create-home --user-group \
		--uid $MKOSI_UID \
		--password '$2a$10$XVhjfOB4Un5DJE4TQEBPrOHfBVGVWP4iA3ElUMzcbJ7jdc2zZPgZ2' \

	## User testing with ssh.
	useradd --create-home --user-group --groups wheel \
		--uid $((MKOSI_UID+1)) \
		--password '$2a$10$XVhjfOB4Un5DJE4TQEBPrOHfBVGVWP4iA3ElUMzcbJ7jdc2zZPgZ2' \

	su - awwan sh -c "mkdir -p .ssh; \
        ssh-keygen -t ed25519 -f .ssh/id_ed25519 -N '' -C awwan@image"
	su - awwanssh sh -c "mkdir -p .ssh"
	cat /home/awwan/.ssh/ \
        > /home/awwanssh/.ssh/authorized_keys
	chown awwanssh:awwanssh /home/awwanssh/.ssh/authorized_keys

I think the script is quite self-explainable. If $container is not "mkosi" we re-execute the script to run inside image using mkosi-chroot. Once the script is run inside chroot and its in "final" state (after all packages are installed), we create user "awwan" with UID similar to current user that run the mkosi and user "awwanssh" with UID+1.

Under user "awwan" we generate new SSH key and copy the public key to second user "awwanssh", so user "awwan" can SSH to "awwanssh" without password prompt.

That’s it. Now we can build our image,

Building image

The image must be build using root privileged,

$ sudo mkosi --directory=_mkosi/

Once the above command completed, we will have one directory and one file created inside _mkosi directory,

├── awwan-test/
├── awwan-test.nspawn


This is the output of our image, in format of directory.


This is the copy of "mkosi.nspawn". This file is required when running "mkosi shell", "mkosi boot", "systemd-nspawn", or "machinectl" later.

Testing and building awwan

First we boot the image. I created a make task to do this,

.PHONY: setup-mkosi
	@echo ">>> Creating symlinks to simplify binding ..."
	ln -sf $(shell go env GOCACHE) _mkosi/mkosi.cache/gocache
	ln -sf $(shell go env GOMODCACHE) _mkosi/mkosi.cache/gomodcache
	@echo ">>> Booting awwan-test container ..."
	sudo mkosi --directory=_mkosi/ boot

When we execute the task,

$ make setup-mkosi
>>> Creating symlinks to simplify binding ...
ln -sf /home/ms/.cache/go-build _mkosi/mkosi.cache/gocache
ln -sf /home/ms/go/pkg/mod _mkosi/mkosi.cache/gomodcache
>>> Booting awwan-test container ...
sudo mkosi --directory=_mkosi/ boot
[sudo] password for ms:
systemd 254.5-1-arch running in system mode (+PAM +AUDIT -SELINUX -APPARMOR
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Received regular credentials: agetty.autologin, firstboot.locale,
firstboot.timezone, login.noauth
Acquired 4 regular credentials, 0 untrusted credentials.

Welcome to Arch Linux!

Failed to open libbpf, cgroup BPF features disabled: Operation not supported
Queued start job for default target Graphical Interface.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Started OpenSSH Daemon.
         Starting User Login Management...
[  OK  ] Started Verify integrity of password and group files.
[  OK  ] Started D-Bus System Message Bus.
         Starting Home Area Manager...
[  OK  ] Started Home Area Manager.
[  OK  ] Finished Home Area Activation.
         Starting Permit User Sessions...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started User Login Management.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.

Arch Linux 6.5.6-arch2-1 (pts/0)

awwan-test login:

The container run and ready to be used.

Then we build the test binary, and run it on container, using the following make task,

.PHONY: test-with-mkosi
	go test -tags=integration -c .
	machinectl shell awwan@awwan-test \
		/bin/sh -c "cd src; ./awwan.test -test.v"

The "go test -tags=integration -c ." means we build the test binary that contains only "//go:build integration" constrains, the output binary file name is "awwan.test". So, when we run the test binary in the container, only the test that have "integration" tags will be executed.

Lets try it,

$ make test-with-mkosi
CGO_ENABLED=1 go test -race -c .
machinectl shell awwan@awwan-test /bin/sh -c "cd src; ./awwan.test"
Authentication is required to acquire a shell in a local container.
Authenticating as: ms
Connected to machine awwan-test. Press ^] three times within 1s to exit
--- BaseDir: /home/awwan/src/testdata/decrypt-with-passphrase
--- BaseDir: /home/awwan/src/testdata/decrypt-with-passphrase
--- Loading private key file ".ssh/awwan.key" (enter to skip passphrase) ...
--- BaseDir: /home/awwan/src/testdata/decrypt-with-passphrase
--- BaseDir: /home/awwan/src/testdata/local
--- Loading "awwan.env" ...
--- Loading ".awwan.env.vault" ...
--- Loading private key file ".ssh/awwan.key" (enter to skip passphrase) ...
Connection to machine awwan-test terminated.