H2 Labs
← What we build and run

UHT · hardened, immutable Linux

Immutable Linux,
prepared for the job.

UHT is how we set up Fedora’s immutable Linux on our workstations and on the hosts that run it. Each machine starts from a signed, read-only base that updates as a whole and keeps the previous version ready to boot, with SELinux enforcing. Our workstations add a hardened kernel, encrypted disks, hardware keys and backups that speak up when they fail. When we set up or harden machines with your team, this is the approach we bring.

One foundation, two roles

Both roles start from the same signed, read-only base, with SELinux enforcing. What goes on top depends on the job the machine does, and on what that job can afford to give up.

Fig. 1 UHT: one foundation, prepared for different jobs.

The workstation is where the access lives: the keys, the administration, the code. It runs the kernel’s strictest lockdown, and most of its desktop apps run as Flatpaks, kept separate from the base system.

The host runs services. A host we monitor keeps eBPF tracing available, so a collector on it sees every connection in and out, live.

Between them, SSH works only with keys that live on a hardware security key, and our hosts accept nothing else.

Layer by layer

What UHT puts on a machine, from the base up, as it runs on our workstations. Every layer is written down with the reason for it, and most come with a check in the uht command.

Table 1 The setup on our workstations, from the base up.

  1. 01 / base

    A signed, read-only base

    Fedora Silverblue. The base comes as one image, signed by Fedora and checked against that signature, and the packages we add are layered onto it as part of the same deployment. The system files stay read-only while the machine runs, and changing them takes a whole new deployment.

  2. 02 / selinux

    SELinux enforcing

    Policy enforced by the kernel limits what services may touch, beyond ordinary file permissions. It is enforcing on our workstations and on the hosts that run UHT, and any exception is written down with its reason.

  3. 03 / kernel

    A hardened kernel

    Four boot arguments. init_on_free=1 wipes memory as it is freed, page_alloc.shuffle=1 hands out memory pages in random order, slab_nomerge keeps kernel caches apart, and vsyscall=none switches off a legacy memory region used in attacks. Lockdown comes on top, set per machine.

  4. 04 / disks

    Encrypted disks

    The drives are encrypted with LUKS: the system disk, apart from its small boot partitions, and both drives of the backup mirror.

  5. 05 / keys

    Hardware keys

    Raising privileges with sudo asks for a touch on a FIDO2 security key, such as a Nitrokey or a YubiKey, with the password kept as the fallback. The SSH keys our hosts accept live on the security key itself, so every new connection to them needs a touch.

  6. 06 / isolation

    Work kept apart

    Most desktop apps run as Flatpaks, kept separate from the base system. Toolboxes for other distributions run in containers, and virtual machines, each with a kernel of its own, are at hand for code we do not trust.

Checked against what is written

The uht command checks a machine against the written setup in 16 areas, from the backups and the hardware keys to the packages and the containers. Most follow a numbered document, and each prints the command that fixes what it finds.

# all 16 areas
uht check

# one area: the backups
uht check backup

One update, one whole system

An update is a complete new system. It waits beside the running one and takes over at the next reboot, and the previous one stays in the boot menu, one step back.

Fig. 2 Staged, running, previous: each a whole system.

Staged, then started. Nothing in the running system changes until the reboot. On our workstations the disk passphrase is typed at boot, so a person is there when the new system starts.

Read before the reboot. rpm-ostree db diff lists what the staged system changes, package by package. One staged change carried 45 upgrades and 14 removals, the screen locker among the removals, and the list showed it before anything took over.

One step back. If the new system misbehaves, the previous one is a choice in the boot menu, or one command: rpm-ostree rollback.

Firmware on its own. A bad flash can stop a machine from booting, so firmware is never part of a system update: a person installs it, at a time chosen for it.

Lockdown, set per machine

Kernel lockdown keeps even root from changing the running kernel, and at its strictest level from reading what the kernel holds. That level also stops eBPF tracing, which monitoring tools use to watch a machine at work. So we choose the level for each machine.

Fig. 3 The three lockdown levels, and where our machines sit.

Our workstations run the strictest level. They are where the access lives, and tracing their kernel is a job we can do without.

A host we monitor runs one level lower. Its running kernel still cannot be changed, and eBPF tracing stays available, so a collector on the host sees every connection in and out of it as it happens.

It is the balance we set together with you for each kind of system, here at the scale of one kernel setting.

(i) Every gain in security costs something. Here the gain is a kernel even root cannot read, the cost is eBPF monitoring, and we choose between them for each machine.

Undo and backup

Two layers on our workstations, each with its own job. Snapshots undo a mistake; encrypted copies on two other drives in the same machine are there for the day a disk gives out.

Undo · same disk

Hourly snapshots

Every hour, a read-only snapshot of the home folders, on the same disk. A file deleted or overwritten by mistake comes back with uht restore, which asks first.

Kept: 10 hourly, 7 daily, 4 weekly, 3 monthly

Backup · other drives

Daily encrypted copies

Every day, the changes since the last copy go to a mirror of two encrypted drives, apart from the disk they come from.

Kept: 14 daily, 12 weekly, 12 monthly

A failure speaks up. When the daily copy fails, a note goes to a phone, naming only the job and the machine; the details stay in the machine’s own log. The backup check, uht check backup, looks at how fresh both layers are as well as how they are set, so a job that has quietly stopped shows up as a failure the next time the check runs.

Harden your machines with us

We bring the same approach to your workstations and servers: one foundation, a setting agreed for each role, and every choice written down with its reason. Tell us what you run, and we answer plainly, including what we would leave as it is.

Talk to us  How we start
  1. LookYour machines as they are. We read how they are set up today, with the people who run them.
  2. AgreeA setting for each role. What is locked down, what stays open for monitoring or for the work, and why.
  3. PrepareHardened with your team. We prepare the machines together, one by one, and the written setup stays with you.