Cannot layer any package adding a user or group on Fedora 42 #5365
Comments
|
I had this same issue when trying to use F42/Cosmic, except for me it was the |
|
This was similarly reported in on the Silverblue tracker - fedora-silverblue/issue-tracker#643 We believe it is an issue with how @jeckersb I'd suggest looking at the |
|
This is a bad update. Fedora 42 Atomic variants can no longer override (which we must do due to systemd-remount-fs.service failure) or update the system to new version. I am facing the same issue as everyone else I understand that this is an upstream issue. But shouldn't it be tested and coordinated with upstream before releasing newer version of rpm-ostree which breaks a critical functionality like system update? It would be great if a workaround is suggested until a fix is made by upstream. |
|
For me, removing the layered qemu package fixes the issue, so the issue description is wrong, not all packages causing the issue, but I am suprised that this issue was not spotted in beta testing. Good thing we can rollback the update thanks to the underlying technologies |
If the
I understand that this is problem is frustrating, howerver I think that testing that every package in Fedora can be successfully layered on top of Silverblue is a large (and possibly unreasonable) ask. Please remember that there are humans on the other end of these issues that are trying their best in their free time to keep all these projects successful. If you want to help out towards the success of these projects, you can provide valuable feedback by testing your use cases on Rawhide or even as part of the next Beta. |
No, this won't work due to this sysusers issue.
I didn't mean in a negative way. Of course I understand all the thankless contributions and efforts behind the project. But this is not the first time rpm-ostree has broken system update. Any application or package specific issue, users can workaround. But breaking system update itself in a final release should have been atleast mentioned in release notes. There are many packages in fedora that uses sysusers (I do not have deep knowledge on that). So rpm-ostree cannot install those packages or allow updating/overriding the system with those packages layered. QEMU is a very commonly used package in Fedora community especially in Atomic variants. So maybe it should have been handled before the release. Anyway I ended up doing
I have done in the past and would be very happy to report in future as well. |
I'm facing this issue when trying to rebase from 41 to 42 kinoite with layered packages. Is there a fundamental workflow change that's supposed to happen here? It isn't addressed in any documentation, as far as I could find, so I would call this an issue still. Any advice? |
|
I wonder if creating the missing group manually (through newgrp) can solve the issue untll https://bugzilla.redhat.com/show_bug.cgi?id=2359764 is solved. DId anybody test that ? |
I tried pulling the entry from I'm not in a place where it feels worth nuking my layered packages to upgrade at this point. If I have to start over, I'll end up on openSuse. |
|
I wonder what is wrong with https://src.fedoraproject.org/rpms/systemtap/blob/rawhide/f/systemtap.spec Hope someone will fix it soon... No ability to deploy security fixes because of a specific package issue for a bit more than 2 weeks is a concern ... |
|
This might not be only an F42 issue, I see the same problem on F41 Sericea, it seems to be |
|
The issue is still present for me as well trying to upgrade from Silverblue 41 to 42... The newest systemtap packages (5.3) did not solve the issue. |
|
This is not upstream issue and has been reassigned back to rpm-ostree package. Please check https://bugzilla.redhat.com/show_bug.cgi?id=2359764 It's not just systemtap, but also wireshark, openvpn Wireshark: OpenVPN: While bugs are expected for an experimental software like Silverblue (and Atomic variants), these new platforms must not be claimed as production ready, as evident by multiple bugs throughout the releases. I have been using Silverblue maybe since it's inception. But the problem is it is advertised in fedoraproject.org as if it is stable. So users expect Silverblue to be stable like Workstation which is clearly not the case. Hence I propose to mark Atomic variants as "Alpha" or "Experimental" so users clearly know what kind of workloads can be put on them. Edit: Other ways to install virt-manager seems very complicated or not working as expected. Hence I have moved back to good old Workstation as my work depends on virt-manager. I hope Atomic variants gets stable and polished like the mainstream Workstation. But It was a good ride for years. Thanks to all the contributors. |
|
I am unable to install openvpn with
on coreos 42. Is there a workaround? Is there likely to be a fix at some point soon? @bitestring ... I have been using Silverblue since its inception, and it has been mostly rocksolid. The issues that it faces from time to time are not because silverblue or other atomic hosts like coreos are in some sort of 'alpha' state. Those issues are specific to the atomic host variant and are simply issues that are going to be encountered from time to time. I use coreos in production and it serves me extremely well, with this current issue, as I am unable to install openvpn, being only the second failure since I began using silverblue and coreos, when they were first released. That this issue is interrupting production servers means that it should be made a priority. In the meantime I am going to have to try reverting to fedora 41, which is not really acceptable, but there seems to be no other choice. |
|
@millerthegorilla This also affects F41, I am on F41 and having the same issue as it has the same rpm-ostree version that causes the problems.
On 14 May 2025 14:13:27 UTC, James Stewart Miller ***@***.***> wrote:
millerthegorilla left a comment (coreos/rpm-ostree#5365)
I am unable to install openvpn with
```error: While applying overrides for pkg openvpn: Could not find group 'openvpn' in group file```
on coreos 42. Is there a workaround?
Is there likely to be a fix at some point soon?
@bitestring ... I have been using Silverblue since its inception, and it has been mostly rocksolid. The issues that it faces from time to time are not because silverblue or other atomic hosts like coreos are in some sort of 'alpha' state. Those issues are specific to the atomic host variant and are simply issues that are going to be encountered from time to time.
The move to systemd-sysusers is non trivial, and was bound to cause some issues.
It would have been nice if this particular issue should have been spotted whilst in rawhide, but as stated above, not every package can be checked.
I use coreos in production and it serves me extremely well, with this current issue, as I am unable to install openvpn, being only the second failure since I began using silverblue and coreos, when they were first released.
That this issue is interrupting production servers means that it should be made a priority. In the meantime I am going to have to try reverting to fedora 41, which is not really acceptable, but there seems to be no other choice. Its a pain as I am using coreos-installer and so I have to manually download the f41 image which is a pain.
--
Reply to this email directly or view it on GitHub:
#5365 (comment)
You are receiving this because you commented.
Message ID: ***@***.***>
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
|
|
I just installed openvpn successfully on my silverblue 42 machine. It required an --allow-inactive to install it though. |
|
@mihalyr I have also just installed openvpn succesfully on the most recent coreos f41. |
|
@mihalyr but once installed, I am unable to install anything else or update. My silverblue 42 install is fine but I think openvpn is in the base image. I am thinking I might learn how to use ostree compose or coreos-assembler to make a base coreos image with openvpn already layered. @jlebon is there any update on this issue? Is there a github issue number or a bugzilla case? |
|
There doesn't seem to be much investigation of this issue going on. I can't find any real evidence of it affecting a large number of people nor can I raise any support from maintainers. So perhaps it is an edge case, only affecting a very small amount of people? |
|
No it is not, I have seen numerous reports of folks not able to upgrade to Silverblue 42 because of systemtap. The latest release (5.3) of systemtap should have solve the issue but it still remains on my end. |
|
Is systemtap the same issue as openvpn? I thought this was an rpm-ostree
issue, due to the change to systemd-sysusers...?
James Stewart Miller Bsc(hons) Psych.
…On Fri, 16 May 2025, 12:07 Jean-Francois Saucier, ***@***.***> wrote:
*djfjeff* left a comment (coreos/rpm-ostree#5365)
<#5365 (comment)>
No it is not, I have seen numerous reports of folks not able to upgrade to
Silverblue 42 because of systemtap. The latest release (5.3) of systemtap
should have solve the issue but it still remains on my end.
—
Reply to this email directly, view it on GitHub
<#5365 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA4GSGPB5T6U7CF7S2RHNY326XBH7AVCNFSM6AAAAAB3IQDWDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQOBWGQYDSNJXHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
travier
changed the title
|
Hello there, I am chiming in to report that the same issue seems to affect the The behavior I can observe while updating a machine with the The machine experiencing this is running Fedora IoT |
|
I think this affects many packages that have migrated to sysusers. But ddclient-4.0.0-1.fc42 can be layerd without any problem. The differences of the two packages are the following two commits, which are basically snippets to migrate to sysusers: |
|
I am guessing that because this issue is only affecting some packages and not all, that the package maintainers have to make adjustments to their installation pre and post scripts to allow them to work with immutable systems now that sysusers has been implemented. |
|
I'm experiencing this same issue with I expect this is a particularly tricky issue to collect feedback on, as there's not much indication that it has to do with rpm-ostree itself; it took me quite a bit of time to find this issue, as I assumed it was simply a packaging problem. |
|
I second how difficult it is to find relevant bug reports/issues. I heartily recommend that anyone who is experiencing this bug opens an issue at fedora bugzilla. |
|
I'm on |
rpm-ostree has issues with installing qemu now: coreos/rpm-ostree#5365
|
OK I'm looking at this now. The sysusers rework in Fedora 42 I think was a generally good idea, but it uncovered a lot of technical debt in this area in rpm-ostree. As for why it works for some packages but not others, I think it basically only errors if the package has content owned by the dynamic user/group. For example with openvpn: Now in general of course this situation is exactly the one being debated heavily in various places (xref bootc-dev/bootc#1263 and the fedora-devel thread) From what I can see initially what's happening here is that the new users are being added to Further findings:
|
cgwalters
added
priority/high
triaged
|
Some preparatory work in #5398 I just want to reiterate a finding from earlier that took me a while to understand: Since #5334 the "pure systemd-sysusers" case works for base image builds solely because there's no altfiles setup at the time, and then later we make all users move to altfiles. For client layering it's a super complex problem though. With classic useradd/groupadd, altfiles is really the only choice, because (unlike sysusers) there's no "reconcile on boot" case. The key question here I think is: in the package layering flow, do we treat sysusers as another way to extend nss-altfiles, or do we try to keep the users in Keeping them in The best fix I think would be to only use sysusers ➡ altfiles for users that own content in the target root. We'd then discard the other sysusers entries, ensuring that they get created on the client system instead as usual (because they don't need to be lifecycle bound to the filesystem tree). And if we go down that route, it becomes most obvious to implement things the same way librpm does, by parsing the sysusers content out of the However there's yet another quite different alternative: We could allow systemd-sysusers to mutate the live (booted) |
|
We can distinguish packages that have "owned files" by looking at rpm metadata. If Requires:user(...) is present, package has files owned by user. If Requires:group(...) is present, package has files owned by the group or has a user belonging to that group. So this should be relatively straightforward to implement. |
What if a package starts off not owning content and then owning it? We'd have to switch it to altfiles, but client-side
I think... as annoying as it is, my vote is to just keep extending nss-altfiles for consistency. It's how the server-side works and keeping that unified core symmetry is valuable. |
|
It might not be entirely relevant to the issue at hand but a problem I had with nss-altfiles is that often I would add a user to a group using I can't claim to understand the complexities of the current situation but if nss-altfiles continues to be the solution does that mean that I am going to have to continue to use 'vigrp' and manually edit the /etc file to satisfy requirements of some programs that don't know about nss-altfiles? As an end user I would much prefer to get anticipated consequences of the usermod command. The current issue may be a different one altogether, handling only installation configuration, but if there is an opportunity to restore a compliant usermod command then I would be super happy. |
|
Hi @millerthegorilla yes it is a relevant point - if we continued to add to nss-altfiles we'd continue to have that issue for groups like
Why would we have to switch it to altfiles?
OK. I am really on the fence myself. The contrary argument is that mutating the live |
Because there would be content in the commit that references it. |
|
The prospect of leaking state presented by mutating the live tree is one issue, but as an end user interested in security, I would far prefer for installed packages to keep their groups in an unwritable location, ie /usr/lib than the potentially writable in comparison /etc. This might not make much sense, as if root were pwned then the system would be compromised beyond repair, but I like the potential security offered by an immutable file system, it's why I chose coreos, and to place groups in /etc would begin to diminish that. As for the issue of programs that are unware of nss-altfiles missing a group in /etc could /etc/group be a symbolic link to /usr/lib (ridiculous perhaps) or nss-altfiles or similar be developed to add all groups in /usr/lib/ to /etc/group? The only solution I had for the earlier nss-altfiles issue was to manually add the group/user to /etc and the net result was that the group/user was in both /usr/lib and /etc simultaneously. It had no problematic side effects. I am probably exposing my lack of understanding of the issue though... |
For now, we'll just treat sysusers entries from RPM packages like we do scriptlets that `useradd`/`groupadd`; that is, we want them to happen at compose time and go into altfiles in case those same sysusers own content in the commit. All we need to do to make that happen is to run `systemd-sysusers` _after_ we do the `/etc/passwd` <--> `/usr/lib/passwd` switcheroo so that the new entries go into what will become `/usr/lib/passwd`. And all we need to do that is to just move down the sysusers execution a bit. Fixes: coreos#5365
I noticed while hacking on coreos#5365 that during the rpmdb writing, librpm was actually re-executing systemd-sysusers *from the host context* which is not at all what we want. Apparently, `RPMTRANS_FLAG_JUSTDB` doesn't imply this and we need to explicitly also pass `RPMTRANS_FLAG_NOSYSUSERS`. That flag doesn't exist in el9, so add a compile-time conditional for it. This fixes the issue for new systems, but people who have upgraded to f42 and overlaid packages with sysusers entries will have new entries in `/etc/passwd` and `/etc/group` files because of this. And this can cause problems now if the UIDs chosen were different because the `/etc` entries will take precedence over nss-altfiles even though owned content will match nss-altfiles. In practice, I think since coreos#5365 breaks exactly those use cases where the sysusers entries own content, we don't have to worry about that subcase. But for sysusers entries that _don't_ own content, the transaction would go through and so there could still be UID conflicts there. I guess we'll need to figure out if to somehow try to fix this or just issue a PSA about it.
Describe the bug
After an update to Fedora Atomic 42, I cannot execute any operation like update, install or uninstall package when I have layered packages.
Reproduction steps
Expected behavior
The operation should complete
Actual behavior
The entries related to this package that I found in the logs shown by
jornalctl -u rpm-ostreed:System details
The issue is also present with version 2025.6
Additional information
No response
The text was updated successfully, but these errors were encountered: