FailedMount and FailedSync events related to builder and deployer tokens #14383
Assignees
Labels
Comments
pweil-
added
component/kubernetes
kind/bug
|
@smarterclayton this looks similar to what @sjenning and i were debugging earlier this week. |
|
we think the problem was introduced in the kube 1.6.1 bump, need to isolate specific commit id. |
|
b282780 is known good and 359f62c is known bad. The commits in between don't build :-/ |
|
What is weird is that I can't reproduce on kube 1.6.0, 1.6.1, 1.6.6, or 1.7 |
|
I found this BZ which was suppose to fix this. https://bugzilla.redhat.com/show_bug.cgi?id=1444096#c4 Even passed QE. https://bugzilla.redhat.com/show_bug.cgi?id=1444096#c18 Supposed to be fixed by #14144 Either 1) it didn't 2) there has been a regression since 3) this is a slightly different but related issue |
|
@ origin commit 22888b2 busybox pod apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "10"
terminationGracePeriodSeconds: 0
restartPolicy: NeverAdditional information from the node log: The following bad events occur: |
|
this is slightly different. My patch was for after a kubelet restart, the events would get spammed because kubelet "forgot" what name to use for a volume. Since there's only one/two events here it's something else |
|
@wongma7 thanks for clarifying |
|
I captured the stray pod worker goroutine that times out after |
|
Added some logging The pod shows as |
|
mistake in my last comment. |
|
Found the difference between kube and origin: Origin unmounts the volumes (last 4 lines) for the terminated pod while kube does not, despite Still don't have an answer for why one unmounts and the other doesn't. I think the expected behavior is unmounting after kubernetes/kubernetes#37228 |
|
This is coarse and very likely has side effects, but does prevent the events from occuring diff --git a/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go b/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go
index e1b7512ee0..b83603821f 100644
--- a/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go
+++ b/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go
@@ -1473,6 +1473,11 @@ func (kl *Kubelet) syncPod(o syncPodOptions) error {
// Update status in the status manager
kl.statusManager.SetPodStatus(pod, apiPodStatus)
+ // Exit syncPod if pod is terminated
+ if kl.podIsTerminated(pod) {
+ return nil
+ }
+
// Kill pod if it should not be running
if !runnable.Admit || pod.DeletionTimestamp != nil || apiPodStatus.Phase == v1.PodFailed {
var syncErr errorIt does seem like much of the rest of |
|
So everything about upstream kube not being effected by this was wrong. I was just running Upstream issue opened. |
This was referenced Jul 27, 2017
k8s-github-robot
pushed a commit
to kubernetes/kubernetes
that referenced
this issue
Aug 1, 2017
Automatic merge from submit-queue (batch tested with PRs 49284, 49555, 47639, 49526, 49724) skip WaitForAttachAndMount for terminated pods in syncPod Fixes #49663 I tried to tread lightly with a small localized change because this needs to be picked to 1.7 and 1.6 as well. I suspect this has been as issue since we started unmounting volumes on pod termination #37228 xref openshift/origin#14383 @derekwaynecarr @eparis @smarterclayton @saad-ali @jwforres /release-note-none
|
it's spam, dropping to p2. (seth is going to open a bz to track in 3.6) |
openshift-merge-robot
added a commit
that referenced
this issue
Aug 2, 2017
Automatic merge from submit-queue UPSTREAM: 49724: skip WaitForAttachAndMount for terminated pods in syncPod Fixes #14383 xref kubernetes/kubernetes#49724 @derekwaynecarr @eparis @smarterclayton @jwforres
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are frequently seeing errors related to mounting secrets as volumes. Commonly appears for the builder-token and deployer-token. Possible this is something only happening on docker for mac. Reproduced by myself and @spadgett so far.
The text was updated successfully, but these errors were encountered: