Golang testing: RunParallel benchmark runs too many iterations

Go version

go version go1.25.1 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/radu_cockroachlabs_com/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/radu_cockroachlabs_com/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4207328522=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/radu_cockroachlabs_com/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/radu_cockroachlabs_com/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/radu_cockroachlabs_com/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I have a standard parallel benchmark:

func BenchmarkFoo(b *testing.B) {
  // <some setup work>
  b.ResetTimer()
  b.RunParallel(func (pb *testing.PB) {
    for pb.Next() {
      // very small amount of work
    }
  }
}

I ran: `go test -run - -bench Foo --benchtime=30s -v .'

What did you see happen?

It does not complete, even after several minutes.

I added a log during the setup work that prints b.N. The verbose output is like this:

goos: linux
goarch: amd64
pkg: github.com/cockroachdb/pebble/internal/cache
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
BenchmarkCacheGet
    cache_test.go:261: N: 1
    cache_test.go:261: N: 69
    cache_test.go:261: N: 4462
    cache_test.go:261: N: 309830
    cache_test.go:261: N: 20830386
    cache_test.go:261: N: 521723665
    cache_test.go:261: N: 521723666
    cache_test.go:261: N: 521723667
    cache_test.go:261: N: 521723668
    cache_test.go:261: N: 521723669
    cache_test.go:261: N: 521723670
    cache_test.go:261: N: 521723671
    cache_test.go:261: N: 521723672
    cache_test.go:261: N: 521723673
    cache_test.go:261: N: 521723674
    cache_test.go:261: N: 521723675
    cache_test.go:261: N: 521723676
    cache_test.go:261: N: 521723677
    cache_test.go:261: N: 521723678
    cache_test.go:261: N: 521723679
    cache_test.go:261: N: 521723680
    cache_test.go:261: N: 521723681
    cache_test.go:261: N: 521723682
...

Clearly there is something wrong with the algorithm that iterates toward finding the "right" N. Which, by the way, shouldn't even be necessary when using pb.Next() (it could in principle behave like b.Loop()).

The machine has 24 CPUs. I can't reproduce this on a different (smaller) machine.

What did you expect to see?

Expected this to take between 30-60s.

Comment From: RaduBerinde

The setup work is non-trivial (maybe around 1 second).

With --benchtime=10s, it works fine:

    cache_test.go:261: N: 1
    cache_test.go:261: N: 21
    cache_test.go:261: N: 428
    cache_test.go:261: N: 9556
    cache_test.go:261: N: 198058
    cache_test.go:261: N: 4414358
    cache_test.go:261: N: 72615568
    cache_test.go:261: N: 240240408
BenchmarkCacheGet-24            240240408               43.75 ns/op

Comment From: RaduBerinde

A simple catch-all would be to stop iterating if the change in b.N is less than say 0.1%

Comment From: mknyszek

CC @adonovan @neild

Comment From: gabyhelp

Related Issues

• testing: strange benchmark results with RunParallel and -benchtime #72079
• testing: benchmark iteration reports incorrectly #41637 (closed)
• testing: run first benchmark with last value from -cpu list GOMAXPROCS value #49980
• testing: Benchmark reporting incorrect stats when running in parallel #28813
• Running testing.Benchmark produces wrong number of iterations #40012 (closed)
• Benchmark results vastly different when Tests are skipped with -run=XXX #42696 (closed)
• expvar: BenchmarkRealworldExpvarUsage times out on specific arm64 server #47511 (closed)
• testing: high start up cost with high -count and no tests #49050 (closed)
• cmd/test2json: benchmark output lines are attributed to the preceding test #49505 (closed)
• cmd/go: use of -bench should force -p=1. #5662 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}