ci-test-infra-triage perma failing

[BenTheElder]

https://prow.k8s.io/?job=ci-test-infra-triage

cc @dims @aojea

/sig testing

timing out at 3h.

[BenTheElder]

We seem to be stuck on data from the 26th now

[BenTheElder]

This was passing reliably until recently:

first failure:

Test started last Saturday at 3:23 PM failed after 3h15m2s
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-triage/1926403800193568768

green before that:

Test started last Saturday at 11:23 AM passed after 2h39m42s.

We don't have a diff link, but it was during 2025-04-25 that it started failing

[BenTheElder]

Something seriously regressed the performance, it was completing in ~50 minutes:

Now taking 3+ hours and timing out:

[BenTheElder]

So maybe something that changed around ~2025-05-16, possibly more data in the CI results or some regression to the tool

[BenTheElder]

#34879 (comment)
So the image with #34876 should've been available by 7:06 AM pacific

A few hours later we are currently running at https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-triage/1928154351960854528

[dims]

@BenTheElder i was watching the previous run, and my change did not solve the issue. i've filed another one #34882 after looking through the logs, let's try that one next. If i don't make progress, will file reverts for both.

[BenTheElder]

@dims your changes sound reasonable to keep IMHO, though I haven't had a chance to read in detail.

Something recent caused this to grow pretty steadily, seems like pathological data. Haven't been able to confirm and I'm out tomorrow (back monday)

[dims]

back on its feet for now: https://prow.k8s.io/?job=ci-test-infra-triage

based on temporary bump(s) for timeout:

• Temporarily bump ci-test-infra-triage timeout/interval #34890
• [triage] bump timeout to 5 hrs for update_summaries script #34892