deployments "should immediately start a new deployment" flake #9681

bparees · 2016-07-02T03:19:25Z

• Failure [31.995 seconds]
deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:501
  when run iteratively
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:182
    should immediately start a new deployment [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:181

    Expected error:
        <*errors.errorString | 0xc8200ce0d0>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    not to have occurred

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:180

as seen in:
https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/2918/consoleFull

The text was updated successfully, but these errors were encountered:

0xmichalis · 2016-07-04T11:57:56Z

Jul  1 22:15:48.006: INFO: At {2016-07-01 22:15:37 -0400 EDT} - event for gentest: {deploymentconfig-controller } DeploymentCancelled: Cancelled deployment "gentest-1" superceded by version 2
Jul  1 22:15:48.006: INFO: At {2016-07-01 22:15:37 -0400 EDT} - event for gentest: {deploymentconfig-controller } DeploymentAwaitingCancellation: Deployment of version 2 awaiting cancellation of older running deployments
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:37 -0400 EDT} - event for gentest: {deploymentconfig-controller } DeploymentCreated: Created new deployment "gentest-1" for version 1
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:37 -0400 EDT} - event for gentest-1-deploy: {default-scheduler } Scheduled: Successfully assigned gentest-1-deploy to 172.18.6.102
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:43 -0400 EDT} - event for gentest-1-deploy: {kubelet 172.18.6.102} Pulled: Container image "openshift/origin-deployer:e01f5cd" already present on machine
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:45 -0400 EDT} - event for gentest-1-deploy: {kubelet 172.18.6.102} Created: Created container with docker id 758b5eaf8f3c
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:46 -0400 EDT} - event for gentest-1-deploy: {kubelet 172.18.6.102} Started: Started container with docker id 758b5eaf8f3c
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:47 -0400 EDT} - event for gentest: { } Started: Running pre-hook ("/bin/echo test pre hook executed") for deployment extended-test-cli-deployment-v85d2-0zt09/gentest-1
Jul  1 22:15:48.007: INFO: At {2016-07-01 22:15:47 -0400 EDT} - event for gentest-1-hook-pre: {default-scheduler } Scheduled: Successfully assigned gentest-1-hook-pre to 172.18.6.102

bparees · 2016-07-07T21:19:54Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3107/consoleFull

0xmichalis · 2016-07-07T21:39:55Z

So it seems that the first deployment is marked as cancelled but the deployer pod is never cleaned up.

bparees · 2016-07-08T17:09:46Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3147/consoleFull

bparees · 2016-07-08T23:57:26Z

[edited: wrong link] https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3166/

bparees · 2016-07-11T14:56:14Z

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3193/consoleFull

0xmichalis · 2016-07-11T15:33:26Z

No need to post logs anymore

On Mon, Jul 11, 2016 at 4:56 PM, Ben Parees [email protected]
wrote:

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3193/consoleFull

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADuFfztGgY2yIByvTUN2ZpgsMH5pmNFlks5qUlmSgaJpZM4JDlvS
.

bparees · 2016-07-11T15:46:27Z

I know you don't need the data but I'm trying to make it clear the
frequency at which this is occurring lest it either get ignored or closed
as "haven't seen this lately".
On Jul 11, 2016 11:33 AM, "Michail Kargakis" [email protected]
wrote:

No need to post logs anymore

On Mon, Jul 11, 2016 at 4:56 PM, Ben Parees [email protected]
wrote:

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3193/consoleFull

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#9681 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe/ADuFfztGgY2yIByvTUN2ZpgsMH5pmNFlks5qUlmSgaJpZM4JDlvS

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEvl3iCUoB9cAxL8wF3HxfI6Ky4EUqDsks5qUmJPgaJpZM4JDlvS
.

0xmichalis · 2016-07-11T15:51:18Z

It's a known issue 2. the frequency of it can be determined by the pulls
it is linked to ie. it is enough to mention it in the thread of occurence.
It won't change anything other than my e-mail count if you post it here. :)

On Mon, Jul 11, 2016 at 5:46 PM, Ben Parees [email protected]
wrote:

I know you don't need the data but I'm trying to make it clear the
frequency at which this is occurring lest it either get ignored or closed
as "haven't seen this lately".
On Jul 11, 2016 11:33 AM, "Michail Kargakis" [email protected]
wrote:

No need to post logs anymore

On Mon, Jul 11, 2016 at 4:56 PM, Ben Parees [email protected]
wrote:

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/3193/consoleFull

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<
#9681 (comment)
,
or mute the thread
<

https://github.com/notifications/unsubscribe/ADuFfztGgY2yIByvTUN2ZpgsMH5pmNFlks5qUlmSgaJpZM4JDlvS

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#9681 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe/AEvl3iCUoB9cAxL8wF3HxfI6Ky4EUqDsks5qUmJPgaJpZM4JDlvS

.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADuFf6F1hfvG1_yDqQ1VXAKAkfT_vdzmks5qUmVYgaJpZM4JDlvS
.

0xmichalis · 2016-07-11T16:19:54Z

STEP: by checking that the second deployment exists
STEP: by checking that the first deployer was deleted and the second deployer exists
Jul 11 00:58:35.167: INFO: DC: &api.DeploymentConfig{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"deployment-simple", GenerateName:"", Namespace:"extended-test-cli-deployment-7z5h0-ajijn", SelfLink:"/oapi/v1/namespaces/extended-test-cli-deployment-7z5h0-ajijn/deploymentconfigs/deployment-simple", UID:"1917f489-4724-11e6-bbac-0e24e71bc3f3", ResourceVersion:"3652", Generation:2, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603809904, nsec:0, loc:(*time.Location)(0x390eaa0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil)}, Spec:api.DeploymentConfigSpec{Strategy:api.DeploymentStrategy{Type:"Rolling", RecreateParams:(*api.RecreateDeploymentStrategyParams)(nil), RollingParams:(*api.RollingDeploymentStrategyParams)(0xc8201c6310), CustomParams:(*api.CustomDeploymentStrategyParams)(nil), Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, Labels:map[string]string(nil), Annotations:map[string]string(nil)}, Triggers:[]api.DeploymentTriggerPolicy{api.DeploymentTriggerPolicy{Type:"ConfigChange", ImageChangeParams:(*api.DeploymentTriggerImageChangeParams)(nil)}}, Replicas:2, Test:false, Paused:false, Selector:map[string]string{"name":"deployment-simple"}, Template:(*api.PodTemplateSpec)(0xc8215ce680)}, Status:api.DeploymentConfigStatus{LatestVersion:2, ObservedGeneration:2, Replicas:2, UpdatedReplicas:0, AvailableReplicas:0, UnavailableReplicas:2, Details:(*api.DeploymentDetails)(0xc820919c80)}}
Jul 11 00:58:35.168: INFO:   RCs: []api.ReplicationController{api.ReplicationController{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"deployment-simple-1", GenerateName:"", Namespace:"extended-test-cli-deployment-7z5h0-ajijn", SelfLink:"/api/v1/namespaces/extended-test-cli-deployment-7z5h0-ajijn/replicationcontrollers/deployment-simple-1", UID:"191baa2e-4724-11e6-bbac-0e24e71bc3f3", ResourceVersion:"3650", Generation:2, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603809904, nsec:0, loc:(*time.Location)(0x390eaa0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"openshift.io/deployment-config.name":"deployment-simple"}, Annotations:map[string]string{"openshift.io/deployment-config.name":"deployment-simple", "openshift.io/deployment.phase":"Failed", "openshift.io/deployment.replicas":"0", "openshift.io/deployment.status-reason":"deployer pod no longer exists", "openshift.io/encoded-deployment-config":"{\"kind\":\"DeploymentConfig\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"deployment-simple\",\"namespace\":\"extended-test-cli-deployment-7z5h0-ajijn\",\"selfLink\":\"/oapi/v1/namespaces/extended-test-cli-deployment-7z5h0-ajijn/deploymentconfigs/deployment-simple\",\"uid\":\"1917f489-4724-11e6-bbac-0e24e71bc3f3\",\"resourceVersion\":\"3566\",\"generation\":1,\"creationTimestamp\":\"2016-07-11T04:58:24Z\"},\"spec\":{\"strategy\":{\"type\":\"Rolling\",\"rollingParams\":{\"updatePeriodSeconds\":1,\"intervalSeconds\":1,\"timeoutSeconds\":600,\"maxUnavailable\":\"25%\",\"maxSurge\":\"25%\"},\"resources\":{}},\"triggers\":[{\"type\":\"ConfigChange\"}],\"replicas\":2,\"test\":false,\"selector\":{\"name\":\"deployment-simple\"},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"name\":\"deployment-simple\"}},\"spec\":{\"containers\":[{\"name\":\"myapp\",\"image\":\"docker.io/openshift/deployment-example:v1\",\"resources\":{},\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8080,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"terminationMessagePath\":\"/dev/termination-log\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"securityContext\":{}}}},\"status\":{\"latestVersion\":1,\"observedGeneration\":1,\"details\":{\"message\":\"caused by a config change\",\"causes\":[{\"type\":\"ConfigChange\"}]}}}\n", "kubectl.kubernetes.io/desired-replicas":"2", "openshift.io/deployer-pod.name":"deployment-simple-1-deploy", "openshift.io/deployment-config.latest-version":"1"}, OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil)}, Spec:api.ReplicationControllerSpec{Replicas:2, Selector:map[string]string{"name":"deployment-simple", "deployment":"deployment-simple-1", "deploymentconfig":"deployment-simple"}, Template:(*api.PodTemplateSpec)(0xc820739380)}, Status:api.ReplicationControllerStatus{Replicas:2, FullyLabeledReplicas:2, ObservedGeneration:2}}, api.ReplicationController{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"deployment-simple-2", GenerateName:"", Namespace:"extended-test-cli-deployment-7z5h0-ajijn", SelfLink:"/api/v1/namespaces/extended-test-cli-deployment-7z5h0-ajijn/replicationcontrollers/deployment-simple-2", UID:"193d46c1-4724-11e6-bbac-0e24e71bc3f3", ResourceVersion:"3633", Generation:3, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603809904, nsec:0, loc:(*time.Location)(0x390eaa0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"openshift.io/deployment-config.name":"deployment-simple"}, Annotations:map[string]string{"openshift.io/deployment-config.latest-version":"2", "openshift.io/deployment-config.name":"deployment-simple", "openshift.io/deployment.phase":"Running", "openshift.io/deployment.replicas":"0", "openshift.io/deployment.status-reason":"caused by a config change", "openshift.io/encoded-deployment-config":"{\"kind\":\"DeploymentConfig\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"deployment-simple\",\"namespace\":\"extended-test-cli-deployment-7z5h0-ajijn\",\"selfLink\":\"/oapi/v1/namespaces/extended-test-cli-deployment-7z5h0-ajijn/deploymentconfigs/deployment-simple\",\"uid\":\"1917f489-4724-11e6-bbac-0e24e71bc3f3\",\"resourceVersion\":\"3579\",\"generation\":2,\"creationTimestamp\":\"2016-07-11T04:58:24Z\"},\"spec\":{\"strategy\":{\"type\":\"Rolling\",\"rollingParams\":{\"updatePeriodSeconds\":1,\"intervalSeconds\":1,\"timeoutSeconds\":600,\"maxUnavailable\":\"25%\",\"maxSurge\":\"25%\"},\"resources\":{}},\"triggers\":[{\"type\":\"ConfigChange\"}],\"replicas\":2,\"test\":false,\"selector\":{\"name\":\"deployment-simple\"},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"name\":\"deployment-simple\"}},\"spec\":{\"containers\":[{\"name\":\"myapp\",\"image\":\"docker.io/openshift/deployment-example:v1\",\"env\":[{\"name\":\"TRY\",\"value\":\"ONCE\"}],\"resources\":{},\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8080,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"terminationMessagePath\":\"/dev/termination-log\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"securityContext\":{}}}},\"status\":{\"latestVersion\":2,\"observedGeneration\":2,\"details\":{\"message\":\"caused by a config change\",\"causes\":[{\"type\":\"ConfigChange\"}]}}}\n", "kubectl.kubernetes.io/desired-replicas":"2", "openshift.io/deployer-pod.name":"deployment-simple-2-deploy"}, OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil)}, Spec:api.ReplicationControllerSpec{Replicas:0, Selector:map[string]string{"deploymentconfig":"deployment-simple", "name":"deployment-simple", "deployment":"deployment-simple-2"}, Template:(*api.PodTemplateSpec)(0xc820739520)}, Status:api.ReplicationControllerStatus{Replicas:0, FullyLabeledReplicas:0, ObservedGeneration:3}}}
Jul 11 00:58:35.168: INFO:   Deployers: map[string][]*api.Pod{"deployment-simple-2":[]*api.Pod{(*api.Pod)(0xc820c786c0)}, "deployment-simple-1":[]*api.Pod{(*api.Pod)(0xc820c78000)}}

The deployer pod printing needs fixing

smarterclayton · 2016-07-11T16:20:57Z

Will try to look at this today.

0xmichalis · 2016-07-11T16:21:27Z

Different test

STEP: by checking that the second deployment exists
Jul  8 12:18:48.534: INFO: DC: &api.DeploymentConfig{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"deployment-simple", GenerateName:"", Namespace:"extended-test-cli-deployment-ohk8o-7bj7q", SelfLink:"/oapi/v1/namespaces/extended-test-cli-deployment-ohk8o-7bj7q/deploymentconfigs/deployment-simple", UID:"a08f9386-4527-11e6-bbe6-0e242ceee0df", ResourceVersion:"5940", Generation:2, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603591518, nsec:0, loc:(*time.Location)(0x3905aa0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil)}, Spec:api.DeploymentConfigSpec{Strategy:api.DeploymentStrategy{Type:"Rolling", RecreateParams:(*api.RecreateDeploymentStrategyParams)(nil), RollingParams:(*api.RollingDeploymentStrategyParams)(0xc820990ee0), CustomParams:(*api.CustomDeploymentStrategyParams)(nil), Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, Labels:map[string]string(nil), Annotations:map[string]string(nil)}, Triggers:[]api.DeploymentTriggerPolicy{api.DeploymentTriggerPolicy{Type:"ConfigChange", ImageChangeParams:(*api.DeploymentTriggerImageChangeParams)(nil)}}, Replicas:2, Test:false, Paused:false, Selector:map[string]string{"name":"deployment-simple"}, Template:(*api.PodTemplateSpec)(0xc8201f24e0)}, Status:api.DeploymentConfigStatus{LatestVersion:2, ObservedGeneration:2, Replicas:0, UpdatedReplicas:0, AvailableReplicas:0, UnavailableReplicas:0, Details:(*api.DeploymentDetails)(0xc820c87b30)}}
Jul  8 12:18:48.534: INFO:   RCs: []api.ReplicationController{api.ReplicationController{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"deployment-simple-1", GenerateName:"", Namespace:"extended-test-cli-deployment-ohk8o-7bj7q", SelfLink:"/api/v1/namespaces/extended-test-cli-deployment-ohk8o-7bj7q/replicationcontrollers/deployment-simple-1", UID:"a0998760-4527-11e6-bbe6-0e242ceee0df", ResourceVersion:"6174", Generation:2, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63603591518, nsec:0, loc:(*time.Location)(0x3905aa0)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"openshift.io/deployment-config.name":"deployment-simple"}, Annotations:map[string]string{"kubectl.kubernetes.io/desired-replicas":"2", "openshift.io/deployment-config.latest-version":"1", "openshift.io/deployment.cancelled":"true", "openshift.io/deployment.replicas":"0", "openshift.io/deployment.phase":"Pending", "openshift.io/encoded-deployment-config":"{\"kind\":\"DeploymentConfig\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"deployment-simple\",\"namespace\":\"extended-test-cli-deployment-ohk8o-7bj7q\",\"selfLink\":\"/oapi/v1/namespaces/extended-test-cli-deployment-ohk8o-7bj7q/deploymentconfigs/deployment-simple\",\"uid\":\"a08f9386-4527-11e6-bbe6-0e242ceee0df\",\"resourceVersion\":\"5927\",\"generation\":1,\"creationTimestamp\":\"2016-07-08T16:18:38Z\"},\"spec\":{\"strategy\":{\"type\":\"Rolling\",\"rollingParams\":{\"updatePeriodSeconds\":1,\"intervalSeconds\":1,\"timeoutSeconds\":600,\"maxUnavailable\":\"25%\",\"maxSurge\":\"25%\"},\"resources\":{}},\"triggers\":[{\"type\":\"ConfigChange\"}],\"replicas\":2,\"test\":false,\"selector\":{\"name\":\"deployment-simple\"},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"name\":\"deployment-simple\"}},\"spec\":{\"containers\":[{\"name\":\"myapp\",\"image\":\"docker.io/openshift/deployment-example:v1\",\"resources\":{},\"readinessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8080,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"terminationMessagePath\":\"/dev/termination-log\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"securityContext\":{}}}},\"status\":{\"latestVersion\":1,\"details\":{\"message\":\"caused by a config change\",\"causes\":[{\"type\":\"ConfigChange\"}]}}}\n", "openshift.io/deployer-pod.name":"deployment-simple-1-deploy", "openshift.io/deployment.status-reason":"newer deployment was found running", "openshift.io/deployment-config.name":"deployment-simple"}, OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil)}, Spec:api.ReplicationControllerSpec{Replicas:2, Selector:map[string]string{"name":"deployment-simple", "deployment":"deployment-simple-1", "deploymentconfig":"deployment-simple"}, Template:(*api.PodTemplateSpec)(0xc8200d51e0)}, Status:api.ReplicationControllerStatus{Replicas:2, FullyLabeledReplicas:2, ObservedGeneration:2}}}

smarterclayton · 2016-07-11T19:05:15Z

Spawned an issue to get deployer pods output

0xmichalis · 2016-07-12T14:57:08Z

A log that includes the deployer output: http://pastebin.com/UNjzbeAi

0xmichalis · 2016-07-12T14:59:18Z

Deletion timestamp on the deployer pod seems unset

DeletionTimestamp:(*unversioned.Time)(0xc820fcddc0)

0xmichalis · 2016-07-12T15:00:46Z

Deletion timestamp on the deployer pod seems unset

Actually it's a pointer, so it is set?

0xmichalis · 2016-07-12T15:02:55Z

It is set... Otherwise it wouldn't show up.

0xmichalis · 2016-07-12T15:07:29Z

We may want to set GracePeriodSeconds when deleting the deployers.

smarterclayton · 2016-07-13T01:07:02Z

It will take some non-zero amount of time to terminate a deployer pod.
Nothing in the controllers should be so dependent on that that they fail.

On Tue, Jul 12, 2016 at 11:00 AM, Michail Kargakis <[email protected]

wrote:

Deletion timestamp on the deployer pod seems unset

Actually it's a pointer, so it is set?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pxKtB9Y1CDRhB7EFvNi-FxyuJTmRks5qU6wggaJpZM4JDlvS
.

mffiedler · 2016-07-13T17:36:41Z

The actual creation of the 2nd rc is much slower in 3.3. If you run this test in 3.2 and oc get rc -o yaml the difference between the first and second rc creation times is always ~3 seconds. On 3.3 it is usually 12 seconds or more.

You can see this by running this manually on 3.2 and 3.3:

oc create -f /root/origin/test/extended/testdata/deployment-simple.yaml;oc set env dc/deployment-simple TRY=ONCE;sleep 5; oc get rc

3.2 always shows the new rc, 3.3 never does. Seems like more of a regression than a flake.

smarterclayton · 2016-07-13T19:34:58Z

Yes.

On Wed, Jul 13, 2016 at 1:36 PM, Mike Fiedler [email protected]
wrote:

The actual creation of the 2nd rc is much slower in 3.3. If you run this
test in 3.2 and oc get rc -o yaml the difference between the first and
second rc creation times is always ~3 seconds. On 3.3 it is usually 12
seconds or more.

You can see this by running this manually on 3.2 and 3.3:

oc create -f /root/origin/test/extended/testdata/deployment-simple.yaml;oc
set env dc/deployment-simple TRY=ONCE;sleep 5; oc get rc

3.2 always shows the new rc, 3.3 never does.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pwlcO-FFXO5cY56zPLZrnBgZ7nu6ks5qVSIrgaJpZM4JDlvS
.

0xmichalis · 2016-07-14T11:09:52Z

I cannot reproduce times north of 5 seconds on master

[vagrant@localhost origin]$ oc create -f test/extended/testdata/deployment-simple.yaml; oc set env dc/deployment-simple test=test; oc get rc -w
deploymentconfig "deployment-simple" created
deploymentconfig "deployment-simple" updated
NAME                  DESIRED   CURRENT   AGE
deployment-simple-1   0         0         1s
NAME                  DESIRED   CURRENT   AGE
deployment-simple-1   0         0         3s
deployment-simple-1   0         0         4s
deployment-simple-2   0         0         0s
deployment-simple-2   0         0         0s
deployment-simple-2   0         0         0s
deployment-simple-2   0         0         3s
^Cdeployment-simple-2   2         0         5s
deployment-simple-2   2         0         5s
deployment-simple-2   2         2         5s
^C[vagrant@localhost origin]$ oc get rc
NAME                  DESIRED   CURRENT   AGE
deployment-simple-1   0         0         15s
deployment-simple-2   2         2         11s

0xmichalis · 2016-07-14T11:17:10Z

The changes between 3.2 and 3.3 for deployments that are subject to a performance hit are the addition of shared caches in the deployment config and generic trigger controller. We also started deep-copying the configs while previously we were mutating the caches (wasn't a big problem since each controller had its own cache). I may have to test 3.2 but in my opinion, pros out of these changes outweight the cons.

0xmichalis · 2016-07-14T11:22:29Z

This test should stop flaking with #9802. Please re-open if you see it.

@smarterclayton @mffiedler please let's move the discussion about the performance regression in a separate issue.

smarterclayton · 2016-07-14T14:18:06Z

The performance issue here is #9775 for secrets.

We should not be deepcopying a deployment without knowing we are going to
mutate it (that makes resync very expensive). So the overly aggressive
copy is safe, but is not an acceptable trade off.

On Jul 14, 2016, at 7:22 AM, Michail Kargakis [email protected]
wrote:

This test should stop flaking with #9802
#9802. Please re-open if you see
it.

@smarterclayton https://github.com/smarterclayton @mffiedler
https://github.com/mffiedler please let's move the discussion about the
performance regression in a separate issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#9681 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_p2Umlyxr_S5BxZBDZ14odiIOawvmks5qVhv3gaJpZM4JDlvS
.

0xmichalis · 2016-07-15T16:21:18Z

We should not be deepcopying a deployment without knowing we are going to mutate it (that makes resync very expensive). So the overly aggressive copy is safe, but is not an acceptable trade off.

Opened #9860

bparees added priority/P2 component/apps area/tests kind/test-flake Categorizes issue or PR as related to test flakes. labels Jul 2, 2016

bparees assigned mfojtik Jul 2, 2016

0xmichalis assigned 0xmichalis and unassigned mfojtik Jul 2, 2016

0xmichalis mentioned this issue Jul 2, 2016

extended: refactor deployment test to be more debuggable #9686

Merged

0xmichalis mentioned this issue Jul 5, 2016

Extended.deploymentconfigs when run iteratively should immediately start a new deployment [Conformance] #9701

Closed

liggitt mentioned this issue Jul 6, 2016

Add debug logging to ldap search bind errors #9706

Merged

bparees mentioned this issue Jul 7, 2016

Ensure client can delete a buildconfig with no associated builds, eve… #9657

Merged

bparees mentioned this issue Jul 8, 2016

bump registry xfs quota; re-enable disk usage diag; add docker info #9753

Merged

danwinship mentioned this issue Jul 11, 2016

Don't let users create endpoints inside ClusterNetworkCIDR #9383

Merged

smarterclayton self-assigned this Jul 11, 2016

smarterclayton mentioned this issue Jul 11, 2016

No info about deployer pods logged #9783

Merged

0xmichalis mentioned this issue Jul 12, 2016

deploy: set gracePeriodSeconds on deployer deletion #9802

Merged

marun mentioned this issue Jul 12, 2016

dind: update to f24 (and go 1.6) #9788

Merged

bparees mentioned this issue Jul 12, 2016

Add mirror env var options and fix required parameters to templates #9754

Merged

bparees mentioned this issue Jul 13, 2016

Refactor stable layer count unit test to get rid of flake #9801

Merged

0xmichalis closed this as completed Jul 14, 2016

xiangpengzhao mentioned this issue Aug 8, 2016

Add more info for test case #10232

Merged

Miciah mentioned this issue Dec 9, 2016

Introduce RestrictUsersAdmission plugin for collaboration #11743

Merged

deployments "should immediately start a new deployment" flake #9681

deployments "should immediately start a new deployment" flake #9681

Comments

bparees commented Jul 2, 2016

0xmichalis commented Jul 4, 2016

Uh oh!

bparees commented Jul 7, 2016

Uh oh!

0xmichalis commented Jul 7, 2016

Uh oh!

bparees commented Jul 8, 2016

Uh oh!

bparees commented Jul 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bparees commented Jul 11, 2016

Uh oh!

0xmichalis commented Jul 11, 2016

Uh oh!

bparees commented Jul 11, 2016

Uh oh!

0xmichalis commented Jul 11, 2016

Uh oh!

0xmichalis commented Jul 11, 2016

Uh oh!

smarterclayton commented Jul 11, 2016

Uh oh!

0xmichalis commented Jul 11, 2016

Uh oh!

smarterclayton commented Jul 11, 2016

Uh oh!

0xmichalis commented Jul 12, 2016

Uh oh!

0xmichalis commented Jul 12, 2016

Uh oh!

0xmichalis commented Jul 12, 2016

Uh oh!

0xmichalis commented Jul 12, 2016

Uh oh!

0xmichalis commented Jul 12, 2016

Uh oh!

smarterclayton commented Jul 13, 2016

Uh oh!

mffiedler commented Jul 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarterclayton commented Jul 13, 2016

Uh oh!

0xmichalis commented Jul 14, 2016

Uh oh!

0xmichalis commented Jul 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xmichalis commented Jul 14, 2016

Uh oh!

smarterclayton commented Jul 14, 2016

Uh oh!

0xmichalis commented Jul 15, 2016

Uh oh!

bparees commented Jul 8, 2016 •

edited

Loading

mffiedler commented Jul 13, 2016 •

edited

Loading

0xmichalis commented Jul 14, 2016 •

edited

Loading