Golang x/build/cmd/golangbuild: generate instructions for reproducing test results

When tests currently fail it's a bit tricky to figure out exactly how to reproduce the environment that created them. We can automatically generate some nice instructions for failing tests in golangbuild and post them somewhere in the build.

Some examples:

# Go to a GOROOT.
cd $(go env GOROOT)

# Download golangbuild.
echo "infra/experimental/golangbuild/linux-amd64 latest" | cipd ensure -root . -ensure-file=-

# Set up the environment. Replace bash with your favorite shell.
GOMOTE_SETUP=gotip-linux-amd64-boringcrypto-test_only ./golangbuild bash

# Run the test.
./bin/go test -run="^MyTest$" net 

# Go to a GOROOT.
cd $(go env GOROOT)

# Create, push, and build toolchain on gomote.
export GOMOTE_GROUP=debug
GOROOT=$PWD gomote create -setup gotip-linux-amd64-boringcrypto

# Run the test.
gomote run go/bin/go test -run="^MyTest$" net 

# Go to a GOROOT.
cd $(go env GOROOT)

# Download golangbuild.
echo "infra/experimental/golangbuild/linux-amd64 latest" | cipd ensure -root . -ensure-file=-

# Set up the environment. Replace bash with your favorite shell.
GOMOTE_SETUP=gotip-js-wasm-test_only ./golangbuild bash

# Run the test.
./bin/go test -run="^MyTest$" net 

# Go to a GOROOT.
cd $(go env GOROOT)

# Create, push, and build toolchain on gomote.
export GOMOTE_GROUP=debug
GOROOT=$PWD gomote create -setup gotip-js-wasm

# Run the test.
gomote run go/bin/go test -run="^MyTest$" net

# Go to a GOROOT.
cd $(go env GOROOT)

# Download golangbuild.
echo "infra/experimental/golangbuild/windows-amd64 latest" | cipd ensure -root . -ensure-file=-

# Set up the environment and run the test.
GOMOTE_SETUP=gotip-windows-amd64-race-test_only ./golangbuild.exe ./bin/go test -run="^MyTest$" net 

# Go to a GOROOT.
cd $(go env GOROOT)

# Create, push, and build toolchain on gomote.
export GOMOTE_GROUP=debug
GOROOT=$PWD gomote create -setup gotip-windows-amd64-race

# Run the test.
gomote run go/bin/go test -run="^MyTest$" net

This text should be fairly straightforward to generate. The tricky part is figuring out where to surface it. Ideally, it would appear next to the test results, but that means result_adapter would need to know how to add in custom additional artifacts per-test, which seems complex to do and requires buy-in. We could fork result_adapter for our needs, which may be worth it.

Alternatively, we surface a link to such instructions in the build results summary (which is what shows up in Gerrit and ends up in people's inboxes). This doesn't require changing result_adapter, but it is a little less discoverable, and may require golangbuild to actually parse and make sense of the Go test JSON format itself, which is also undesirable.

Comment From: mknyszek

Alternatively, one could imagine an subcommand to the gomote command called repro that takes a build link and figures everything out from there.

What this option might look like to implement: * Extend the GOMOTE_SETUP mode for golangbuild to accept a LUCI build ID. * If golangbuild sees a build ID instead of a builder in GOMOTE_SETUP, it fetches the right versions of any repositories in addition to setting up the environment and tools. It also makes sure all the Go toolchains involves are built (but this comes for free, pretty much). * gomote repro <build id> becomes really simple: it creates a gomote for the right builder and invokes GOMOTE_SETUP=<build id> golangbuild (golangbuild is available on the machine). * From there, gomote could fetch the build info itself and print some useful next steps based on the build's test results, among other things.

We get a two-for-one from here. Now, if someone wants to reproduce the environment on their own machine, they can use GOMOTE_SETUP=<build id> golangbuild bash whereas if someone wants the full environment, or doesn't have local access to a GOOS/GOARCH pair, then gomote does that for you.

Comment From: mknyszek

I landed a CL adding GOMOTE_REPRO to golangbuild which accepts a build ID, to avoid parsing ambiguities. I sent a CL for gomote repro <build ID> as well.

Comment From: gopherbot

Change https://go.dev/cl/601440 mentions this issue: cmd/gomote: move valid builder list out of default help text for create

Comment From: gopherbot

Change https://go.dev/cl/601438 mentions this issue: internal/gomote: allow subrepo builder types

Comment From: gopherbot

Change https://go.dev/cl/601437 mentions this issue: cmd/gomote: add repro subcommand

Comment From: mknyszek

All the above CLs have landed, and I just landed a CL to golangbuild to suggest a gomote repro command for failing builds. I think this can be closed now.