Allow not evicting pods that belong to unknown controllers

[grosser]

Description

What problem are you trying to solve?
We have pods that belong to custom controllers, that should not be considered replaceable.
Atm we have to add "do not disrupt" annotation to them via admission,
but it would be nice to have a feature that protects these pods (either by default or optional)
similar to what cluster-autoscaler does with --skip-nodes-with-custom-controller-pods (docs) which is default true

How important is this feature to you?
Not a dealbreaker, but surprising when switching from cluster-autoscaler

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

Can you elaborate on what it means for a pod to be owned by a custom controller? Does this mean the pod has an owner-reference for something other than a replica or daemonset?

/triage needs-information

[grosser]

For example we have an operator that creates "user consoles" that users will exec into to run tasks,
they will be owned by the operator and should not be interrupted.
In general I think it makes sense to not assume that every pod owned by a unknown controller is replaceable (or at least make it an option).