Golang testing/synctest: Repeated sync.WaitGroup.Add appears flaky under synctest

Go version

go version go1.25rc1 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/alexhamlin/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/alexhamlin/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/1v/jwmc7lqn4sv37y_yppfyr32c0000gn/T/go-build478068048=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/alexhamlin/Desktop/synctest-waitgroup-go/go.mod'
GOMODCACHE='/Users/alexhamlin/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/alexhamlin/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/alexhamlin/sdk/go1.25rc1'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/Users/alexhamlin/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/alexhamlin/sdk/go1.25rc1/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25rc1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

https://go.dev/play/p/4itIC1JGITi?v=gotip

Alternatively, go test -count=10000 with the case below that runs fewer iterations in the inner loop. go test -race -count=1000 also reliably reproduces on my M1 MacBook Pro.

More observations:

• Further synchronization in the goroutine (e.g. incrementing a counter under a mutex) also makes reproduction more reliable.
• go version go1.25-devel_b5d555991a on darwin/arm64 reproduces as well (when I make the go.mod + GOTOOLCHAIN updates to ensure it's really used).
• linux/arm64 (Pine64 ROCKPro64) and linux/amd64 (random VPS host) reproduce as well.
• It almost goes without saying, but sync.WaitGroup.Go reproduces this just as well as the classic Add + Done variant shown below (in fact, my attempt to use the new API is how I noticed this).
• I'm not able to reproduce this with a single wg.Add(100) before entering the loop.

module github.com/ahamlinman/synctest-waitgroup-go

go 1.25rc1

package main

import (
    "sync"
    "testing"
    "testing/synctest"
)

func TestSynctestWaitGroupGo(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        var wg sync.WaitGroup
        for range 100 {
            wg.Add(1)
            go func() {
                defer wg.Done()
            }()
        }
        wg.Wait()
    })
}

What did you see happen?

One of the following two kinds of crashes.

panic: sync: negative WaitGroup counter

goroutine 808574 [running, synctest bubble 7850]:
sync.(*WaitGroup).Add(0x14000380cc0, 0xffffffffffffffff)
        /Users/alexhamlin/sdk/go1.25rc1/src/sync/waitgroup.go:118 +0x288
sync.(*WaitGroup).Done(...)
        /Users/alexhamlin/sdk/go1.25rc1/src/sync/waitgroup.go:158
github.com/ahamlinman/synctest-waitgroup-go.TestSynctestWaitGroupGo.func1.1()
        /Users/alexhamlin/Desktop/synctest-waitgroup-go/main_test.go:16 +0x48
created by github.com/ahamlinman/synctest-waitgroup-go.TestSynctestWaitGroupGo.func1 in goroutine 805551
        /Users/alexhamlin/Desktop/synctest-waitgroup-go/main_test.go:14 +0x38
exit status 2
FAIL    github.com/ahamlinman/synctest-waitgroup-go     0.530s

fatal error: sync: WaitGroup.Add called from inside and outside synctest bubble

goroutine 236056 [running, synctest bubble 2886]:
sync.fatal({0x1021e8fe3?, 0x1021d20a8?})
        /Users/alexhamlin/sdk/go1.25rc1/src/runtime/panic.go:1026 +0x20
sync.(*WaitGroup).Add(0x14000182050, 0x1)
        /Users/alexhamlin/sdk/go1.25rc1/src/sync/waitgroup.go:100 +0xe8
github.com/ahamlinman/synctest-waitgroup-go.TestSynctestWaitGroupGo.func1(0x14000246380?)
        /Users/alexhamlin/Desktop/synctest-waitgroup-go/main_test.go:13 +0x58
testing.tRunner(0x14000246380, 0x10225d370)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:1931 +0xc8
created by testing/synctest.testingSynctestTest in goroutine 236055
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2043 +0x1fc

goroutine 1 [chan receive]:
testing.(*T).Run(0x14000246000, {0x1021e126a?, 0x1400007ab38?}, 0x10225d2e0)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2002 +0x378
testing.runTests.func1(0x14000246000)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2474 +0x38
testing.tRunner(0x14000246000, 0x1400007ac68)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:1931 +0xc8
testing.runTests(0x1400000c030, {0x102348c80, 0x1, 0x1}, {0x140000681e0?, 0x7?, 0x102352a20?})
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2472 +0x3b8
testing.(*M).Run(0x1400007e140)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2334 +0x530
main.main()
        _testmain.go:45 +0x80

goroutine 236055 [chan receive (durable), synctest bubble 2886]:
testing/synctest.testingSynctestTest(0x140002461c0, 0x10225d370)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:2044 +0x210
testing/synctest.Test.func1()
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/synctest/synctest.go:283 +0x28
created by testing/synctest.Test in goroutine 236054
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/synctest/synctest.go:282 +0x88

goroutine 236054 [synctest.Run (durable), synctest bubble 2886]:
internal/synctest.Run(0x14000356020)
        /Users/alexhamlin/sdk/go1.25rc1/src/runtime/synctest.go:218 +0x1b4
testing/synctest.Test(0x140002461c0, 0x10225d370)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/synctest/synctest.go:282 +0x88
github.com/ahamlinman/synctest-waitgroup-go.TestSynctestWaitGroupGo(0x140002461c0?)
        /Users/alexhamlin/Desktop/synctest-waitgroup-go/main_test.go:10 +0x24
testing.tRunner(0x140002461c0, 0x10225d2e0)
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:1931 +0xc8
created by testing.(*T).Run in goroutine 1
        /Users/alexhamlin/sdk/go1.25rc1/src/testing/testing.go:1994 +0x364
exit status 2
FAIL    github.com/ahamlinman/synctest-waitgroup-go     0.328s

What did you expect to see?

A passing test, given that the entire sync.WaitGroup and all operations on it are contained in a single synctest bubble.

Comment From: gabyhelp

Related Issues

• testing: inconsistent behaviors between running tests directly and after compiling the code first #59879 (closed)
• sync: apparent deadlock in TestWaitGroupMisuse3 #35774 (closed)
• golang.org/x/sync/errgroup: strange data race #65045 (closed)
• affected/package: sync #50544 (closed)
• sync.WaitGroup panic trouble me #39013 (closed)
• sync: Once panics if testing.T.FailX or testing.T.SkipX are called #73159
• affected/package: sync #53769 (closed)
• sync: wait group produces data race when waiting before adding in separate goroutine #56728 (closed)
• sync: random errors on sync.Once running on MacOS Mojave or High Serra #30453 (closed)

Related Code Changes

• sync: deflake TestWaitGroupMisuse2

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: neild

We've got a bug in disassociating WaitGroups from bubbles.

Right now, we associate a WaitGroup with a bubble on the first Add call, and disassociate it from the bubble when an Add call causes its count to go to zero. When the disassociation races with another Add call, we can end up with inconsistent state.

The simplest fix would be to not disassociate WaitGroups from bubbles--once one is in a bubble, it stays there. This might not even require a documentation change; I think the ability to move a WaitGroup between bubbles (so long as it has a count of zero and no waiters) is undocumented.

Or possibly we can fix the disassociation. The current approach is obviously wrong now that I look at it, and not trivial to fix, but it might be possible.

Comment From: gopherbot

Change https://go.dev/cl/684635 mentions this issue: sync: disassociate WaitGroups from bubbles on Wait