Golang x/build: perf builders are failing with "infra failed"

Both gotip-linux-amd64_c2s16-perf_vs_release and gotip-linux-amd64_c3h88-perf_vs_release are failing with "infra failed"

Error in build protocol: Expected a terminal build status, got STATUS_UNSPECIFIED, while top level status is STARTED.

E.g. https://ci.chromium.org/ui/p/golang/builders/ci/gotip-linux-amd64_c2s16-perf_vs_release/b8742123808151649825/overview

The failure has persisted for 5 commits.

cc @golang/release

Comment From: cherrymui

cc @mknyszek

Comment From: gabyhelp

Related Issues and Documentation

• build: build failure on gotip-linux-amd64_c2s16-perf_vs_release #68437

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: dmitshur

In the last successful build from 20 hours ago, the "7. upload benchmark results" step took 6.9 seconds. In the failing builds since then, it's failing after 7-9 minutes and the "results" log appears very large (e.g., around 1.4 GB).

Comment From: mknyszek

I know the root cause and I have a CL out. I'll mark that CL with this issue. In short, yeah, the log is just huge because CockroachDB spews a ton of data out and a recent x/benchmarks CL dumped that data by default.

Comment From: mknyszek

The CL is https://go.dev/cl/599455 if anyone wants to give it a quick review.

Comment From: gopherbot

Change https://go.dev/cl/599455 mentions this issue: sweet: collect and dump only hijacked fd 2 for cockroachdb