run.sh does not execute automatically, preventing airgap setup from completing

[jarrillaga]

Hey community :)

I'm building an airgap ISO using AuroraBoot and a valid OCI bundle.tar that contains:

• /assets/k3s-airgap-images-amd64.tar.gz
• /assets/run.sh (script to copy the image into /var/lib/rancher/k3s/agent/images/)
• config.yaml with stages.boot

I also pass an external config.yaml to AuroraBoot, which is identical to the one inside the bundle (same content), but with different intention:

• The external one is meant to control installation (install: and stages.install)
• The internal one is just to define stages.bootthat will run after first boot

What works:

• System installs and reboots properly.
• External config is applied (e.g. user creation works).
• stages.boot from the bundle runs and writes to /usr/local/.state/kairos-stage.log.

What doesn’t work:

• run.sh (inside the bundle) is never executed.
• stages.install from the external config.yaml is silently skipped, even though install: works.

I unpacked the OCI with umoci in my local and confirmed everything is in place. It seems Kairos prioritizes the config from the bundle for stages and ignores stages.install from the external config, even though it processes the install: block without executing run.sh but considering the config because of stages.boot.

Questions:

What’s the clean, official way to execute the run.sh from the bundle to move the k3s image to a persistent path?

Why is stages.install skipped with no warning?

Why doesn’t the airgap example specify this clearly about how run.sh is executed?

Thanks!

[Itxaka]

I don't think we have a stages.install. we have a stage called kairos-install.pre and kairos-install.post

Maybe it's that?

[jarrillaga]

@Itxaka really appreciate your quick answe, that could be it, my mistake for assuming stages.install existed without checking thoroughly. I’ll give kairos-install.post a try and report back.

What still feels a bit strange is that run.sh inside the bundle.tar is not executed automatically after the bundle is mounted. Given that the airgap example doesn’t mention manually invoking it, I assumed it would be handled implicitly.

Thanks again!:

install:
  auto: true
  device: "auto"
  reboot: true
  bundles:
    - targets:
        - run:///run/initramfs/live/bundle.tar
      local_file: true

[Itxaka]

@Itxaka really appreciate your quick answe, that could be it, my mistake for assuming stages.install existed without checking thoroughly. I’ll give kairos-install.post a try and report back.

What still feels a bit strange is that run.sh inside the bundle.tar is not executed automatically after the bundle is mounted. Given that the airgap example doesn’t mention manually invoking it, I assumed it would be handled implicitly.

Thanks again!:

install:
  auto: true
  device: "auto"
  reboot: true
  bundles:
    - targets:
        - run:///run/initramfs/live/bundle.tar
      local_file: true

Tbh I have zero idea about how bundles work, I'll bring this up tomorrow with the team to see if someone can answer your questions better than me :)

[richardelling]

In the install phase, the destination of /var/lib/rancher/k3s/agent/images/ is on the installer's overlay, which is not where you want the files. You want the files in the /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/ directory. So you have 2 choices:

1. move the run bundle target from install.targets (during install) to bundles.targets (after first boot, before starting k3s)
2. change the destination directory in your run.sh -- this is arguably better because it will work for either case in option 1

To better understand the ordering, kairos-init runs the k3s installer, which creates systemd services and copies some other files around. The creation of the k3s cluster occurs when the systemd k3s.service is started by the provisioner-kairos, after first boot. For airgap, you need the container images file k3s-airgap-images-amd64.tar.gz to be already present in /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/, which is the directory bind mounted onto /var/lib/rancher at boot time.

The silence you hear from the run.sh is because stdout is ignored and stderr is only shown when there is a non-zero exit code from run.sh. I've been meaning to propose an improvement for this to help with debugging bundles. In a pinch, you can manually store run.sh logs in a persistent location, rather than relying on stdout.

[jarrillaga]

Thanks a lot @richardelling and @Itxaka — your answers helped me get the K3s image copied successfully to the right location.

However, once the image was finally in place, K3s started failing on boot.

Interestingly, in all my previous tests (where the image wasn't copied correctly), K3s would start without any visible errors. Now that the image is present, I’m seeing this:

Active: activating (auto-restart) (Result: exit-code)
ExecStart=/usr/bin/k3s server (code=exited, status=1/FAILURE)
Main PID: ... (code=exited, status=1/FAILURE)

My suspicion is that the example copies the image as k3s-airgap-images-amd64.tar.gz, but K3s expects a .tar file, not a gzipped archive — and there's no step that decompresses it.

Also, I’d like to share some findings and a few questions. Hopefully, this helps clarify a few edge cases.

1. bundle.tar format is not fully OCI-compliant

The bundle.tar created via docker build + docker save looks like an OCI layout (blobs/, index.json, etc.), but can’t be unpacked with umoci:
⨯ open CAS: validate: read oci-layout: invalid image detected

I tried to generate a bundle with umoci, replicating the Kairos structure (run.sh, assets/k3s-airgap-images-amd64.tar.gz), but Kairos didn’t detect or run it.

Is docker build + docker save the only supported method to create a valid bundle.tar?

2. Possible issue in the docs: asset path

In the example Dockerfile, we have:

COPY --from=alpine /build/k3s-airgap-images-amd64.tar.gz /assets

This rename k3s image to assets, rather than placing it in an asset directory, and the run.sh is not consistent with this.

In summary, run.sh executes automatically, even without explicitly defining a stage in config.yaml that call it, as long as the example configuration is followed correctly, particularly by generating bundle.tar with docker build + docker save, and ensuring the paths inside the image (especially the asset and script locations) are correct.

[richardelling]

We grab the ztsd compressed version, .tar.zstd because it is smaller. It loads fine.

The ctr command is part of the default installation, so we debug with ctr -n k8s.io -a /run/k3s/containerd/containerd.sock mount $IMAGE $TMPDIR

We also use buildah to create containers, mostly because it does not require a container runtime. But the build shouldn't really matter. The resulting container can be inspected in several ways, including nerdctl verify

For the Dockerfile COPY command, the destination needs to be consistent. The strangeness of docker COPY is another reason we prefer buildah, but it could also be your trouble. The destination directory needs to exist, so you have two choices:

1. COPY --from=alpine /build/ /assets/
2. Explicitly create the destination directory:

RUN umask=022 && mkdir /assets
COPY --from=alpine --chmod=444 /build/k3s-airgap-images-amd64.tar.gz /assets/

Again, you can debug by mounting the container with ctr as shown above and then inspecting the container and running the run.sh script.

Pro tip: the run.sh script is passed to sh -- the Bourne shell, not BASH. So when you test your run.sh do the same:

TMPDIR=$(mktemp -d)
ctr -n k8s.io -a /run/k3s/containerd/containerd.sock mount $IMAGE $TMPDIR
cd $TMPDIR
sh -x run.sh

The -x option is useful for debugging scripts.

To help prevent ourselves from accidentally adding bash-isms into run.sh we add a step in the CI/CD pipeline to shellcheck -s sh run.sh

[jarrillaga]

Hey guys, continuing with the K3s airgap topic, after copying the packed .gz image to the persistent directory where Kairos expects it, I found that my initial assumption was correct: it's absolutely necessary to run gunzip -f to extract the .tar file at that location. Without this step, K3s won’t start properly.

This step is currently missing from the documentation, and I believe it's a key issue, along with the path inconsistencies I mentioned earlier. The fix itself is simple, just adding a gunzip -f line to the run.sh script solves the problem completely.

Another thing worth pointing out is that the default size of the persistent partition is only 250MB, which makes it impossible to unpack the K3s image or store any other binaries or artifacts required in an airgap setup. This needs to be explicitly defined in the config.yaml using the partitions.persistent.size field, but this requirement isn't mentioned in the airgap docs either.

Honestly, getting to this point took quite a few days of debugging, even though, on paper, it should be relatively straightforward if the documentation were clearer.

Here’s the link to the example: https://kairos.io/docs/examples/airgap/

I really think it's worth revisiting this part of the docs to save others the same troubleshooting time and also clarify if docker build + docker save is the unique accepted method to create the bundle.tar

Happy to contribute if needed.

[Itxaka]

Another thing worth pointing out is that the default size of the persistent partition is only 250MB, which makes it impossible to unpack the K3s image or store any other binaries or artifacts required in an airgap setup. This needs to be explicitly defined in the config.yaml using the partitions.persistent.size field, but this requirement isn't mentioned in the airgap docs either.

? What image are you using for this? Persistent by default is set to 0 size, which means that it will take as much space possible during the install, so if you have a 30Gb disk it will take all free space after creating the other partitions, and same with a 3Tb disk.

Something must be wrong if persistent is by default 250Mb!!

The rest of the comment, I agree with, we need a cleanup on the airgap stuff. We been putting it off as we were getting to the systemd-sysext and wanted to see if we could bridge the gap and use them for this, but we should probably update this sooner than later

[jarrillaga]

@Itxaka I'm using the same image that's in the official example: https://github.com/k3s-io/k3s/releases/download/v1.23.16+k3s1/k3s-airgap-images-amd64.tar.gz

I'm not sure why the persistent partition ended up being 250MB, I don't think my hard drive only had that much space available during the test. But after I added the partitions.persistent.size field and set it to 10GB, everything worked like a charm.

[richardelling]

k3s uses the github.com/rancher/warfie project to load images into the container runtime. Supported filename extensions are
".tar", ".tar.lz4", ".tar.bz2", ".tbz", ".tar.gz", ".tgz", ".tar.zst", ".tzst" so perhaps something else was at play? It is trivial to test because you can just copy a file there and use something like nerdctl image ls to see if the container gets automatically loaded.

Reference
https://github.com/rancher/wharfie/blob/main/pkg/tarfile/tarfile.go#L25

[tbrasser]

Just for reference, I'm trying a full offline install of k0s, here is my current cloud-config and Dockerfile. The node installs, boots, k0s starts, kubelet is posting node ready, and kube-proxy pod is running. :)

It required quite some more digging than I had anticipated, but it's a good learning journey. Although maybe both the k0s and k3s instructions can do with a bit of an update. (Maybe a proper reference for the k0s and k3s sections in cloud-config would help!).

config.tpl.yaml (I envsubst to config.yaml before using with AuroraBoot)

#cloud-config

hostname: node-1

users:
  - name: kairos
    passwd: ${USERPASS}
    groups:
      - admin
  # ssh-authorized-keys:
  #   - github:${GITHUB_USER} 
install:
  auto: true
  reboot: true
  device: ${KAIROS_DEVICE}
  no-format: false
  force: true
stages:
  initramfs:
    - files:
        - path: /etc/systemd/network/01-dummy.netdev
          permissions: 0644
          content: |
            [NetDev]
            Name=dummy0
            Kind=dummy
        - path: /etc/systemd/network/02-dummy.network
          permissions: 0644
          content: |
            [Match]
            Name=dummy0
            [Network]
            Address=${IP}
        - path: /etc/k0s/k0s.yaml
          permissions: 0644
          content: |
            apiVersion: k0s.k0sproject.io/v1beta1
            kind: ClusterConfig
            metadata:
              name: k0s
            spec:
              api:
                address: ${IP}
        - path: /etc/systemd/system/k0scontroller.service.d/override.conf
          permissions: 0644
          content: |
            [Service]
            ExecStart=
            ExecStart=/usr/bin/k0s controller --enable-worker --no-taints
  boot:
    - name: "Start k0s"
      if: '[ -e "/sbin/systemctl" ] || [ -e "/usr/bin/systemctl" ] || [ -e "/usr/sbin/systemctl" ] || [ -e "/usr/bin/systemctl" ]'
      commands:
        - systemctl start k0scontroller
p2p:
  disable_dht: true  # discover nodes only in the local network
  network_token: ${EDGEVPN_TOKEN}
  auto:
    enable: true
    ha:
      enable: true
      master_nodes: 2  # first node is implied
k0s:
  enabled: true

Dockerfile

ARG BASE_IMAGE=ubuntu:24.10
ARG VARIANT=standard
ARG MODEL=generic
ARG TRUSTED_BOOT=false
ARG KUBERNETES_DISTRO=k0s
ARG KUBERNETES_VERSION=v1.33.1
ARG KUBEVIRT_VERSION=v1.6.0-beta.0
ARG FRAMEWORK_VERSION=v2.22.0
ARG VERSION=0.0.1

FROM quay.io/kairos/kairos-init:latest AS kairos-init

FROM alpine AS alpine
ARG KUBERNETES_VERSION
ARG KUBEVIRT_VERSION
RUN wget https://github.com/k0sproject/k0s/releases/download/${KUBERNETES_VERSION}%2Bk0s.0/k0s-airgap-bundle-${KUBERNETES_VERSION}+k0s.0-amd64
RUN wget https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-operator.yaml

FROM ${BASE_IMAGE} AS base-kairos
ARG KUBERNETES_VERSION
ARG FRAMEWORK_VERSION
ARG MODEL
ARG VARIANT
ARG TRUSTED_BOOT
ARG KUBERNETES_DISTRO
ARG VERSION
COPY --from=kairos-init /kairos-init /kairos-init
RUN mkdir -p /var/lib/k0s/images /var/lib/k0s/manifests
COPY --from=alpine /k0s-airgap-bundle-${KUBERNETES_VERSION}+k0s.0-amd64 /var/lib/k0s/images/image-bundle.tar
COPY --from=alpine /kubevirt-operator.yaml /var/lib/k0s/manifests/kubevirt-operator.yaml
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug -s install -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}+k0s.0" --version "${VERSION}"
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug -s init -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}+k0s.0" --version "${VERSION}"
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug --validate -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}+k0s.0" --version "${VERSION}"
RUN rm /kairos-init