Golang testing: early break in a testing.B.Loop benchmark loop should be reported.

Go version

go version go1.24.1 darwin/arm64

Output of `go env` in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/shaojunyang/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/shaojunyang/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/6t/d5y6mrc90rzd1gm2q2k99lb400zg0c/T/go-build3077995788=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/Users/shaojunyang/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/shaojunyang/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/shaojunyang/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

For benchmark that breaks with its own logic, -benchtime won't work and the users should be notified.

What did you see happen?

If we run the following benchmark with go test -bench=. -benchtime=1s:

func BenchmarkFoo(b *testing.B) {
        i := 0
        for b.Loop() {
                fmt.Printf("iter %d\n", i)
                i += 1
        if i > 5 {
            break
        }
                time.Sleep(100 * time.Millisecond)
        }
}

The output would be

iter 0
iter 1
iter 2
iter 3
iter 4
iter 5
goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkFoo-10           10      50555750 ns/op
PASS
ok      command-line-arguments  0.732s

What did you expect to see?

Instead of allowing the benchmark to pass with bogus numbers, we should let the user know that the benchmark failed.

Comment From: gopherbot

Change https://go.dev/cl/659655 mentions this issue: testing: separate b.Loop counter from b.N