WIP: Build via Containerfile and derive from fedora-bootc #3348

jlebon · 2025-02-03T20:23:05Z

This builds Fedora CoreOS using a Containerfile. It uses the FROM scratch flow to do a base compose.

This is structured in a way that we can build FCOS using both podman build and cosa build. This allows us to make the cutover much smoother. So then, we could turn this on in e.g. rawhide first and let it percolate down.

All the heavy lifting is done in the build-rootfs script. The idea is that this script is shared by both FCOS and RHCOS. Random notes:

It's in Python. This is fine, because the builder image is separate from the target rootfs. The bootc-base-imagectl script itself is in Python and so can only be used from the :standard image.
Packages are extracted from the manifest, and fed to bootc-base-imagectl (see
https://gitlab.com/fedora/bootc/base-images/-/merge_requests/178) so it's part of a single compose.
We keep respecting the in-git lockfiles. In the future, how lockfiles work will change (see discussions in Build FCOS using podman build fedora-coreos-tracker#1861).
What you see overall is a lot of cosa-isms and rpm-ostree-isms being lifted and carried right into the build script. Things like overlays, versioning, postprocessing, os-release mutating, etc...

Rechunking is expected to be done as a secondary step. Once we have containers/buildah#5952, we can inline it back into the Containerfile.

Requires:

ship bootc-base-imagectl in Fedora (#44) (currently to test this, you need to build fedora-bootc locally yourself)
compose: Cache RPMs if cache dir on separate filesystem rpm-ostree#5391
Some bits of base-imagectl: support --cachedir; use in Containerfile (!180)

jlebon · 2025-02-03T20:24:37Z

This is still in the early phase. It doesn't work yet.

Also note we're using tier-1 for now since there's no tier-x image available yet (https://gitlab.com/fedora/bootc/base-images/-/issues/25).

Edit: now using a tier-x image I built myself.

jlebon · 2025-02-07T18:30:04Z

OK cool, got a successful build with this and coreos/rpm-ostree#5274 which I can even rebase to and boot successfully. We're not rebuilding the initramfs yet nor adding overlays. Those shouldn't be hard though.

jlebon · 2025-05-20T19:32:05Z

Updated this now and updated the PR description with more details! Still working on it (notably, going over preliminary diffs between a cosa-built tree and this).

This builds Fedora CoreOS using a Containerfile. It uses the [FROM scratch flow](https://docs.fedoraproject.org/en-US/bootc/building-from-scratch/) to do a base compose. This is structured in a way that we can build FCOS using _both_ `podman build` _and_ `cosa build`. This allows us to make the cutover much smoother. So then, we could turn this on in e.g. rawhide first and let it percolate down. All the heavy lifting is done in the `build-rootfs` script. The idea is that this script is shared by both FCOS and RHCOS. Random notes: 1. It's in Python. This is fine, because the builder image is separate from the target rootfs. The `bootc-base-imagectl` script itself is in Python and so can only be used from the `:standard` image. 2. Packages are extracted from the manifest, and fed to `bootc-base-imagectl` (see https://gitlab.com/fedora/bootc/base-images/-/merge_requests/178) so it's part of a single compose. 3. We keep respecting the in-git lockfiles. In the future, how lockfiles work will change (see discussions in coreos/fedora-coreos-tracker#1861). 4. What you see overall is a lot of cosa-isms and rpm-ostree-isms being lifted and carried right into the build script. Things like overlays, versioning, postprocessing, os-release mutating, etc... Rechunking is expected to be done as a secondary step. Once we have containers/buildah#5952, we can inline it back into the Containerfile.

cgwalters

Looks generally sane to me!

cgwalters · 2025-05-27T18:30:26Z

image-default.yaml

+# This file contains defaults for image.yaml
+
+bootfs: "ext4"
+rootfs: "xfs"


Can you also link here osbuild/bootc-image-builder#932

dustymabe · 2025-05-29T20:48:12Z

Containerfile

+# # XXX: for hacking
+COPY rpm-ostree /usr/bin/
+COPY bootc-base-imagectl /usr/libexec/
+RUN --mount=type=cache,rw,id=cache,target=/cache \


Suggested change

RUN --mount=type=cache,rw,id=cache,target=/cache \

RUN --mount=type=cache,rw,id=coreos-build-cache,target=/cache \

dustymabe · 2025-05-29T20:51:21Z

build-rootfs

suggestion.. to reduce confusion, since this is coreos specific for now, rename to coreos-build-rootfs.

dustymabe · 2025-05-29T20:59:09Z

build-rootfs

+            argsfile.write("--install\n")
+            argsfile.write(f"{pkg}\n")


Suggested change

argsfile.write("--install\n")

argsfile.write(f"{pkg}\n")

argsfile.write(f"--install\n{pkg}\n")

or would argsfile.write(f"--install={pkg}\n") work?

dustymabe · 2025-05-29T21:02:10Z

build-rootfs

+    # inject live/ bits
+    target_path = os.path.join(rootfs, 'usr/share/coreos-assembler/live')
+    shutil.copytree(os.path.join(CONTEXTDIR, "live"), target_path)


I guess we could put this in a proper overlay to not have to special case this.

dustymabe · 2025-05-29T21:19:27Z

manifests/fedora-coreos-base.yaml

@@ -102,6 +100,26 @@ postprocess:
    ConditionPathExists=
    EOF

+  # sudo prefers its config files to be mode 440, and some security scanners
+  # complain if /etc/sudoers.d files are world-readable.
+  # https://bugzilla.redhat.com/show_bug.cgi?id=1981979


Should we mention what "overlay" this is from?

dustymabe · 2025-05-29T21:19:54Z

manifests/fedora-coreos-base.yaml

+    set -xeuo pipefail
+    ln -sf multi-user.target /usr/lib/systemd/system/default.target
+
+  - |


Would be nice to have a comment for this block.

dustymabe · 2025-05-29T21:20:56Z

manifests/fedora-coreos-base.yaml

+    set -xeuo pipefail
+    if [ -f /run/.containerenv ]; then
+      grep sudo /usr/lib/group >> /etc/group
+    fi


Should we add these three new postprocess scripts in a manifest that is shared with RHCOS/SCOS?

dustymabe · 2025-05-29T21:22:06Z

manifests/fedora-coreos.yaml

+    if [ -f /run/.containerenv ]; then
+      conf_file=/etc/rpm-ostreed.conf
+    fi
+


Maybe delete this newline

dustymabe · 2025-05-29T21:23:35Z

manifests/fedora-coreos.yaml

@@ -134,13 +144,27 @@ postprocess:
        done
    done

+  - |


Would be nice to have a comment for this block.

dustymabe · 2025-05-29T21:24:32Z

manifests/fedora-coreos.yaml

+    rm -f /etc/alternatives/*-man
+
+  # This replicates `etc-group-members` above but for the container-native case.
+  # We can nuke that other one once that's the only path we support.


The comment for the other one is more comprehensive :)

dustymabe · 2025-05-29T21:26:08Z

Havne't had a chance to test this out yet. Some comments in the mean time.

jlebon force-pushed the pr/derive branch from 2076662 to 9d087ee Compare February 7, 2025 18:03

jlebon mentioned this pull request Feb 7, 2025

Add new treefile-apply experimental command coreos/rpm-ostree#5274

Merged

travier mentioned this pull request Mar 14, 2025

Build FCOS using podman build coreos/fedora-coreos-tracker#1861

Open

jlebon force-pushed the pr/derive branch from 9d087ee to 469b300 Compare May 20, 2025 17:00

jlebon changed the base branch from rawhide to testing-devel May 20, 2025 17:00

jlebon force-pushed the pr/derive branch from 469b300 to 5f38573 Compare May 20, 2025 17:01

jlebon force-pushed the pr/derive branch from 5f38573 to 5c5f576 Compare May 23, 2025 18:01

jlebon force-pushed the pr/derive branch from 5c5f576 to 1ee6ffd Compare May 26, 2025 19:02

cgwalters reviewed May 27, 2025

View reviewed changes

dustymabe reviewed May 29, 2025

View reviewed changes

	RUN --mount=type=cache,rw,id=cache,target=/cache \
	RUN --mount=type=cache,rw,id=coreos-build-cache,target=/cache \

	argsfile.write("--install\n")
	argsfile.write(f"{pkg}\n")
	argsfile.write(f"--install\n{pkg}\n")

@@ @@ -134,13 +144,27 @@ postprocess: @@
                       done
                   done
+                - |

WIP: Build via Containerfile and derive from fedora-bootc #3348

Are you sure you want to change the base?

WIP: Build via Containerfile and derive from fedora-bootc #3348

Uh oh!

Conversation

jlebon commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlebon commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlebon commented Feb 7, 2025

Uh oh!

jlebon commented May 20, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dustymabe commented May 29, 2025

Uh oh!

Uh oh!

jlebon commented Feb 3, 2025 •

edited

Loading

jlebon commented Feb 3, 2025 •

edited

Loading