prow.k8s.io job history page showing months-stale results

[BenTheElder]

What happened:

go from: https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default&width=20

to this recent run: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce/1889411351940435968

click the "Job History" link:

https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce

Observe that while the run was from today (feb 11) the job history page is showing results from December.

I suspect a bad metadata file?

I tried https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce?buildId=1889411351940435968 with no success

What you expected to happen:

Job history page should start with recent results.

How to reproduce it (as minimally and precisely as possible):

see above

Please provide links to example occurrences, if any:

see above

Anything else we need to know?:

I haven't noticed this on any other jobs yet

/sig testing k8s-infra

[BenTheElder]

(was looking into kubernetes/kubernetes#129848)

[BenTheElder]

/area prow

[BenTheElder]

$ gsutil cat gs://kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce/latest-build.txt
189489678293716582

But the page starts at 1868710627107672064 from december.

Testgrid goes up to 1894896782937165824 as the currently running job.

[BenTheElder]

Huh, when I refresh I'm getting different results at https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce

Each subsequent load has had newer results. I'm up to mid january ...

[BenTheElder]

It's not quite reliably loading newer runs on subsequent loads but they have averaged newer, They're reliably at least the current month now (still weeks old). Very strange ...

[Prucek]

kubernetes-sigs/prow#388

[BenTheElder]

Thanks! The last few times we've had issues with prow.k8s.io we've eventually root-caused them to migrating the CI to SIG K8s Infra and configuration issues etc so I didn't want to jump to filing a prow bug this time :-)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[BenTheElder]

/remove-lifecycle stale
still true:
https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce

(also it's pretty slow to load initially, which makes me think it's not reading the latest-build.txt https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce/latest-build.txt

manually using the latest build.txt value works, but is not obvious:
https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce?buildId=1928149319215484928