Skip to content

Can't use descheduler.alpha.kubernetes.io/evict=false to disable evicton #1659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kreeuwijk opened this issue Mar 28, 2025 · 6 comments · May be fixed by #1673
Open

Can't use descheduler.alpha.kubernetes.io/evict=false to disable evicton #1659

kreeuwijk opened this issue Mar 28, 2025 · 6 comments · May be fixed by #1673
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@kreeuwijk
Copy link

Is your feature request related to a problem? Please describe.
We use VirtualMachine preferences in Kubevirt to default the use of the descheduler for any VMs using the those presets, by setting the descheduler.alpha.kubernetes.io/evict: 'true' annotation. But this makes it difficult to turn off the descheduler for specific VMs, as the descheduler actually accepts any value for the annotation. Overriding the annotation from the preset through the VM YAML is only possible if you specify an alternate value.

Describe the solution you'd like
I would like to be able to set the annotation

descheduler.alpha.kubernetes.io/evict: 'false'

and have the descheduler treat that pod as if no annotation was set on it.

Describe alternatives you've considered
The only other options that worked were:

  • Remove the annotation from the VM preferences altogether and go back to specifying it in the VM YAML. This isn't very user friendly
  • Create "No DRS" copies of the VM preferences that don't have the annotation but are the same otherwise. This also isn't very user friendly

What version of descheduler are you using?

descheduler version: 0.32.2

Additional context
N/A

@kreeuwijk kreeuwijk added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 28, 2025
@ingvagabund
Copy link
Contributor

ingvagabund commented May 12, 2025

Hello @kreeuwijk. Thank you for opening this issue. Historically, descheduler.alpha.kubernetes.io/evict annotation was used to simplify/shortwire the pre eviction checks/tests. Before the list of eviction policies was extended with other "disable/enable" checks.

and have the descheduler treat that pod as if no annotation was set on it.

There are three possible cases for the annotation if extended that way:

  • descheduler.alpha.kubernetes.io/evict is not set at all: the default behavior of going through all checks: this can either leading into eviction of skiping the pod
  • descheduler.alpha.kubernetes.io/evict: true: ignore all the eviction policy checks: this will result in eviction a pod unless the either of the limits is reached
  • descheduler.alpha.kubernetes.io/evict: false: do not evict a pod

Which is not compatible with your use cases.

We use VirtualMachine preferences in Kubevirt to default the use of the descheduler

Would you please elaborate more on this? There are other ways to tell the descheduler to focus on particular pods. E.g. specifying a label selector or a list of included/excluded namespace. Have you explored this option?

@kreeuwijk
Copy link
Author

kreeuwijk commented May 16, 2025

We use VirtualMachine preferences in Kubevirt to default the use of the descheduler

Would you please elaborate more on this? There are other ways to tell the descheduler to focus on particular pods. E.g. specifying a label selector or a list of included/excluded namespace. Have you explored this option?

Here's an example of a VM preference:

apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineClusterPreference
metadata:
  name: generic-linux
spec:
  annotations:
    descheduler.alpha.kubernetes.io/evict: "true"
  clock:
    preferredClockOffset:
      utc: {}
      #timezone: America/New_York
  cpu:
    preferredCPUTopology: spread   
    spreadOptions:
      across: SocketsCores
      ratio: 4
  devices:
    preferredDiskBus: virtio
    preferredDiskCache: writethrough
    preferredInterfaceModel: virtio
    preferredRng: {}
  machine:
    preferredMachineType: q35
  preferredTerminationGracePeriodSeconds: 180

When you then create a VM, you can link to this VM preference by setting:

spec:
  preference:
    name: generic-linux

in the VirtualMachine object. This is a nice way to reduce the content in the VM spec and make things generally easier on the user. For example, the user doesn't need to set the descheduler annotation on the VM manually, instead it comes via the VM preference.

There is however one issue, that is when the VM preference is used, but the user wants to override the descheduler option and prevent the descheduler from managing this VM. There is no way to remove an annotation coming from a VM preference, but you can override the value of an annotation.

@ingvagabund
Copy link
Contributor

What's the motivation for using the descheduler.alpha.kubernetes.io/evict annotation? Can it be omitted?

@kreeuwijk
Copy link
Author

I tried omitting it, but when the annotation doesn't exist on a VirtualMachine object, it will not get processed by descheduler.

@ingvagabund
Copy link
Contributor

Is there a specific log line that rejects eviction of the VM object? If you run the descheduler with a log level 4 you should see log lines starting with Pod fails the following checks.

@kreeuwijk
Copy link
Author

From https://kubevirt.io/user-guide/compute/node_assignment/#node-balancing-with-descheduler:

Kubevirt VMs are handled as pods with local storage, so by default, descheduler will not evict them.

This is why the annotation is necessary to get the descheduler to process VMs at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
2 participants