Golang testing: b.Loop gives bogus results in some situations

Go version

go version go1.24.0 linux/amd64

What did you do?

There are at least two user errors that testing.B.Loop could detect, but that currently lead to bogus benchmark results instead:

• Returning from a benchmark function before b.Loop() has returned false.
• Stopping the timer in the benchmark loop without starting it again.

What did you see happen?

Here are two examples:

package main

import "testing"

func BenchmarkBreak(b *testing.B) {
    for i := 0; b.Loop(); i++ {
        if i == 2 {
            break
        }
    }
}

func BenchmarkStop(b *testing.B) {
    for i := 0; b.Loop(); i++ {
        b.StopTimer()
    }
}

With Go 1.24 (and tip as of this writing), these produce bogus results:

BenchmarkBreak-8         100             5.950 ns/op
BenchmarkStop-8     458912020            0.0000006 ns/op

With the fix for #72922, BenchmarkStop actually loops forever, until the whole test binary times out.

What did you expect to see?

I expected to see useful errors, and certainly not a timed out benchmark.

Comment From: aclements

cc @JunyangShao

I have some CLs to fix this, which I'll send shortly.

Comment From: gabyhelp

Related Issues

• testing: b.StopTimer breaks b.Loop #72922
• testing: early break in a testing.B.Loop benchmark loop should be reported. #72933
• testing: tiny benchmark with StopTimer runs forever #27217
• testing: add testing.B.Loop for iteration #61515 (closed)

Related Documentation

• Package testing > type B ¶ > func (*B) Loop ¶ 1.24

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: aclements

Good gaby. This is indeed a partial duplicate of https://github.com/golang/go/issues/72933.

Comment From: gopherbot

Change https://go.dev/cl/659657 mentions this issue: testing: detect a stopped timer in B.Loop

Comment From: gopherbot

Change https://go.dev/cl/659655 mentions this issue: testing: separate b.Loop counter from b.N

Comment From: gopherbot

Change https://go.dev/cl/659656 mentions this issue: testing: detect early return from B.Loop

Comment From: aclements

@gopherbot, please open a backport to Go 1.24.

I'm on the fence about whether this should be backported. It makes b.Loop, which is new in Go 1.24, more robust to user mistakes, but those same mistakes could have been made with the old b.N style benchmarks, we just couldn't catch them. In support of backporting, it is a brand new feature and we're going to post a blog post about it soon so it'll probably get a spike in usage.

Comment From: gopherbot

Backport issue(s) opened: #72974 (for 1.24).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.