Release umoci 0.5.0 -- "A wizard is never late, Frodo Baggins. Nor is he early; he arrives precisely when he means to." · opencontainers/umoci

This is a long-awaited release of umoci containing some Go API breaking
changes, some new features, and many other minor changes and
improvements.

Note that the Go API is still considered to be unstable, so downstream
users should generally be aware that future updates may contain more
breaking changes until we release umoci v1.0.0. However, the umoci CLI
is considered to be stable (as it has been widely used for nearly a
decade now) and we will endeavour to not make breaking changes.

This version of umoci requires Go 1.23 to build.

Security

A security flaw was found in the OCI image-spec, where it is possible to
cause a blob with one media-type to be interpreted as a different media-type.
As umoci is not a registry nor does it handle signatures, this vulnerability
had no real impact on umoci but for safety we implemented the now-recommended
media-type embedding and verification. CVE-2021-41190

Breaking

The method of configuring the on-disk format and MapOptions in
RepackOptions and UnpackOptions has been changed. The on-disk format is
now represented with the OnDiskFormat interface, with DirRootfs and
OverlayfsRootfs as possible options to use. MapOptions is now configured
inside the OnDiskFormat setting, which will require callers to adjust their
usage of the main umoci APIs. In particular, examples like
```
unpackOptions := &layer.UnpackOptions{
    MapOptions: mapOptions,
    WhiteoutMode: layer.StandardOCIWhiteout, // or layer.OverlayFSWhiteout
}
err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)
```
will have to now be written as
```
unpackOptions := &layer.UnpackOptions{
    OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
        MapOptions: mapOptions,
    },
}
err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)
```
and similarly
```
repackOptions := &layer.RepackOptions{
    MapOptions: mapOptions,
    TranslateOverlayWhiteouts: false, // or true
}
layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)
```
will have to now be written as
```
repackOptions := &layer.RepackOptions{
    OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
        MapOptions: mapOptions,
    },
}
layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)
```
Note that this means you can easily re-use the OnDiskFormat configuration
between both UnpackOptions and RepackOptions, removing the previous need
to translate between WhiteoutMode and TranslateOverlayWhiteouts.

For users of the API that need to extract the MapOptions from
UnpackOptions and RepackOptions, there is a new helper MapOptions which
will help extract it without doing interface type switching. For
OnDiskFormat there is also a Map method that gives you the inner
MapOptions regardless of type.
layer.NewTarExtractor now takes *UnpackOptions rather than
UnpackOptions to match the signatures of the other layer.* APIs. Passing
nil is equivalent to passing &UnpackOptions{}.
In umoci 0.4.7, we added support for overlayfs unpacking using the
still-unstable Go API. However, the implementation is still missing some key
features and so we will now return errors from APIs that are still missing
key features:
- layer.UnpackManifest and layer.UnpackRootfs will now return an error
  if UnpackOptions.OnDiskFormat is set to anything other than DirRootfs
  (the default, equivalent to WhiteoutMode being set to
  OCIStandardWhiteout in umoci 0.4.7).
  
  This is because bundle-based unpacking currently tries to unpack all
  layers into the same rootfs and generate an mtree manifest -- this
  doesn't make sense for overlayfs-style unpacking and will produce garbage
  bundles as a result. As such, we expect that nobody actually made use of
  this feature (otherwise we would've seen bug reports complaining about it
  being completely broken in the past 4 years). opencontainers/umoci#574
  tracks re-enabling this feature (and exposing to umoci CLI users, if
  possible).
  
  Note that layer.UnpackLayer still supports OverlayfsRootfs
  (OverlayFSWhiteout in umoci 0.4.7).
- Already-extracted bundles with OverlayfsRootfs (OverlayFSWhiteout in
  umoci 0.4.7) will now return an error when umoci operates on
  them -- we included the whiteout mode in our umoci.json but as the
  feature is broken, umoci will now refuse to operate on such bundles. Such
  bundles could only have been created using the now-error-inducing
  UnpackRootfs and UnpackManifest APIs mentioned above, and as mentioned
  above we expect there to have been no real users of this feature.
  
  Note that this only affects extracted bundles (a-la umoci unpack).
  Images created from such bundles are unaffected (even though their
  contents probably should be audited, since the implementation of this
  feature was quite broken in this usecase).
Users should expect more breaking changes in the overlayfs-related Go APIs in
a future umoci 0.6 release, as there is still a lot of work left to do.

Added

umoci unpack now supports handling layers compressed with zstd. This is
something that was added in image-spec v1.2 (which we do not yet support
fully) but at least this will allow users to operate on zstd-compressed
images, which are slowly becoming more common.
umoci repack and umoci insert now support creating zstd-compressed
layers. The default behaviour (called auto) is to try to match the last
layer's compression algorithm, with a fallback to gzip if none of the layer
algorithms were supported.
- Users can specify their preferred compression algorithm using the new
  --compress flag. You can also disable compression entirely using
  --compress=none but --compress=auto will never automatically choose
  none compression.
GenerateLayer and GenerateInsertLayer with OverlayfsRootfs
(called TranslateOverlayWhiteouts in umoci 0.4.7) now support
converting trusted.overlay.opaque=y and trusted.overlay.whiteout
whiteouts into OCI whiteouts when generating OCI layers.
OverlayfsRootfs now supports compatibility with the userxattr mount
option for overlayfs (where user.overlay.* xattrs are used rather than
the default trusted.overlay.*). This is a pretty key compatibility feature
for users that use unprivileged overlayfs mounts and will hopefully remove
the need for most downstream forks hacking in this functionality (such as
stacker). For Go API users, to enable this just set UserXattr: true in
OverlayfsRootfs. Note that (as with upstream overlayfs), only one xattr
namespace is ever used (so if OverlayfsRootfs.UserXattr == true then
trusted.overlay.* xattrs will be treated like any other non-overlayfs
xattr).

Changes

In this release, the primary development branch was renamed to main.
The runtime-spec version of the config.json version we generate is no
longer hard-coded to 1.0.0. We now use the version of the spec we have
imported (with any -dev suffix stripped, as such a prefix causes havoc with
verification tools -- ideally we would only ever use released versions of the
spec but that's not always possible). #452
Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on
cgroupv2 systems.
umoci has been migrated away from github.com/pkg/errors to Go stdlib error
wrapping.
The gzip compression block size has been updated to be more friendly with
Docker and other tools that might round-trip the layer blob data (causing the
hash to change if the block size is different). #509

Fixed

In 0.4.7, a performance regression was introduced as part of the
VerifiedReadCloser hardening work (to read all trailing bytes) which would
cause walk operations on images to hash every blob in the image (even blobs
which we couldn't parse and thus couldn't recurse into). To resolve this, we
no longer recurse into unparseable blobs. #373 #375 #394
Handle EINTR on io.Copy operations. Newer Go versions have added more
opportunistic pre-emption which can cause EINTR errors in io paths that
didn't occur before. #437
Quite a few changes were made to CI to try to avoid issues with fragility.
#452
umoci will now return an explicit error if you pass invalid uid or gid values
to --uid-map and --gid-map rather than silently truncating the value.
For Go users of umoci, GenerateLayer (but not GenerateInsertLayer) with
OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci
0.4.7) had several severe bugs that made the feature unusable:
- All OCI whiteouts added to the archive would incorrectly have the full host
  name of the path rather than the correctly rooted path, making the whiteout
  practically useless.
- Any non-whiteout files would not be included in the layer, making the layer
  data incomplete and thus resulting in silent data loss.
  Given how severe these bugs were and the lack of bug reports of this issue in
  the past 4 years, it seems this feature has not really been used by anyone (I
  hope...).
For Go users of umoci, UnpackLayer now correctly handles several aspects of
OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) extraction
that weren't handled correctly:
- Unlike regular extractions, overlayfs-style extractions require us to
  create the parent directory of the whiteout (rather than ignoring or
  assuming the underlying path exists) because the whiteout is being created
  in a separate layer to the underlying file. We also need to make sure that
  opaque whiteout targets are directories.
- trusted.overlay.opaque=y has very peculiar behaviour when a regular
  whiteout (i.e. mknod c 0 0) is placed inside an opaque directory -- the
  whiteout-ed file appears in readdir but the file itself doesn't exist. To
  avoid this confusion (and possible information leak), umoci will no longer
  extract plain whiteouts within an opaque whiteout directory in the same
  layer. (As per the OCI spec requirements, this is regardless of the order
  of the opaque whiteout and the regular whiteout in the layer archive.)
UnpackLayer and Generate(Insert)Layer now correctly handle
trusted.overlay.* xattr escaping when extracting and generating layers with
the overlayfs on-disk format. This escaping feature has been supported by
overlayfs since Linux 6.7, and
allows for you to created images that contain an overlayfs layout inside the
image (nested to arbitrary levels).
- If an image contains trusted.overlay.* xattrs, UnpackLayer will
  rewrite the xattrs to instead be in the trusted.overlay.overlay.*
  namespace, so that when merged using overlayfs the user will see the
  expected xattrs.
- If an on-disk overlayfs directory used with Generate(Insert)Layer
  contains escaped trusted.overlay.overlay.* xattrs, they will be rewritten
  so that the generated layer contains trusted.overlay.* xattrs. If we
  encounter an unescaped trusted.overlay.* xattr they will not be included
  in the image (though they may cause the file to be converted to a whiteout
  in the image) because they are considered to be an internal aspect of the
  host on-disk format (i.e. trusted.overlay.origin might be automatically
  set by whatever tool is using the overlayfs layers).
  Note that in the regular extraction mode, these xattrs will be treated like
  any other xattrs (this is in contrast to the previous behaviour where they
  would be silently ignored regardless of the on-disk format being used).
When extracting a layer, umoci unpack would previously return an error if a
tar entry was within a non-directory. In practice such cases are quite
unlikely (as layer diffs would usually include an entry changing the type of
the non-directory parent) but this could result in spurious errors with
somewhat non-standard tar archive layers. Now, umoci will remove the
offending non-directory parent component and re-create the parent path as a
proper directory tree.
- This also has the side-effect of fixing the behaviour when unpacking
  whiteouts with the OverlayfsRootfs on-disk format. If there is a plain
  whiteout of a regular directory, followed by parent components being made
  underneath that directory, then the directory should be converted to an
  opaque whiteout. This matches the behaviour of overlayfs (though again, it
  seems unlikely that a layer diff tool would generate such a layer).
  #546

Thanks to all of the following contributors for making this release
possible:

AdamKorcz [email protected]
Aleksa Sarai [email protected]
Bibhas [email protected]
Cameron Nemo [email protected]
Michael McCracken [email protected]
Ramkumar Chinchani [email protected]
Serge Hallyn [email protected]
Shengjing Zhu [email protected]
Tycho Andersen [email protected]
guoguangwu [email protected]

Signed-off-by: Aleksa Sarai [email protected]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

umoci 0.5.0 -- "A wizard is never late, Frodo Baggins. Nor is he early; he arrives precisely when he means to."