#!watchflakes
default <- pkg == "golang.org/x/net/quic" && test == "TestStreamTransfer"
Issue created automatically to collect these failures.
Example (log):
=== RUN TestStreamTransfer
endpoint_test.go:50: read data mismatch (got 1048576 bytes, want 1048576
--- FAIL: TestStreamTransfer (0.05s)
Comment From: gopherbot
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "golang.org/x/net/quic" && test == "TestStreamTransfer"
2025-08-07 18:37 x_net-gotip-windows-amd64 net@af6926ea go@1f7ffca1 x/net/quic.TestStreamTransfer (log)
=== RUN TestStreamTransfer endpoint_test.go:50: read data mismatch (got 1048576 bytes, want 1048576 --- FAIL: TestStreamTransfer (0.05s)Comment From: dmitshur
CC @neild.
Comment From: gopherbot
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "golang.org/x/net/quic" && test == "TestStreamTransfer"
2025-08-07 19:56 x_net-gotip-windows-amd64 net@e74bc31d go@9763ece8 x/net/quic.TestStreamTransfer (log)
=== RUN TestStreamTransfer endpoint_test.go:50: read data mismatch (got 1048576 bytes, want 1048576 --- FAIL: TestStreamTransfer (0.02s)Comment From: neild
I have spent a while looking at this.
The failure reproduces easily on windows-arm64; it doesn't always happen, but -count=100 is usually enough to get a failure. I have not been able to reproduce on any non-Windows platform.
This is, so far as I can tell, a real failure. The test creates a QUIC connection over a loopback socket and sends 1MiB of data from the client to the server. The test fails because the server receives different data than was written to the client connection. The changed data shows up as four bytes unexpectedly converted to what looks like a small integer, usually in several nearby locations in the stream. For example:
endpoint_test.go:53: got: [offset 15488] b2f0efd7 d367e877 ceb2bcf6 53c3a4fb 10000000 1c81014d ee10b39d b0fd2898
endpoint_test.go:54: want: [offset 15488] b2f0efd7 d367e877 ceb2bcf6 53c3a4fb 12bb6306 1c81014d ee10b39d b0fd2898
endpoint_test.go:53: got: [offset 15520] aa81d016 c799a275 00000000 dcda24b6 34bd4a80 180e7f81 00000000 ddf38d7a
endpoint_test.go:54: want: [offset 15520] aa81d016 c799a275 5c47da42 dcda24b6 34bd4a80 180e7f81 9ae3bbb1 ddf38d7a
The exact location in the stream varies, and this doesn't happen every time.
Instrumenting the code, it seems that the contents of the client's send buffer is being modified. The send buffer contains the expected data after the initial Stream.Write call, but has changed at the time the client generates a packet to send.
I have not been able to figure out what is clobbering the send buffer. It shouldn't be modified by anything after the initial Write. I can't find any plausible mechanism by which it would be modified, either--there's only one Write in the entire test, I can verify that the send buffer contains the correct data after that Write, and subsequent access to the send buffer is read-only. If the send buffer was reused, I'd expect it to be more thoroughly clobbered than having a few, non-contiguous, 4-byte sections overwritten.
At this point, I'm wondering if there's a bug unrelated to the quic package resulting in memory clobbering.
Comment From: rsc
Poked at this a little. This makes it much crashier, and by crashier I mean it dies in the implementation of select with an index out of range panic, suggesting serious memory corruption.
diff --git a/quic/endpoint_test.go b/quic/endpoint_test.go
index 98b8756..3aea180 100644
--- a/quic/endpoint_test.go
+++ b/quic/endpoint_test.go
@@ -8,6 +8,7 @@ import (
"bytes"
"context"
"crypto/tls"
+ "fmt"
"io"
"log/slog"
"net/netip"
@@ -29,6 +30,13 @@ func TestConnectDefaultTLSConfig(t *testing.T) {
}
func TestStreamTransfer(t *testing.T) {
+ for i := range 1000 {
+ t.Run(fmt.Sprint(i), testStreamTransfer)
+ }
+}
+
+func testStreamTransfer(t *testing.T) {
+ t.Parallel()
ctx := context.Background()
cli, srv := newLocalConnPair(t, &Config{}, &Config{})
data := makeTestData(1 << 20)
On arm64:
% GOOS=windows GOARCH=arm64 go test -run=TestStreamTransfer
fatal error: index out of range
goroutine 910 gp=0x4000c3a700 m=6 mp=0x4000080808 [running]:
runtime.throw({0x7ff7873b6814?, 0x7ff786fedbd4?})
/Users/rsc/go/src/runtime/panic.go:1227 +0x34 fp=0x4002855ae0 sp=0x4002855ab0 pc=0x7ff787025134
runtime.panicCheck1(0x7ff7876fcd40?, {0x7ff7873b6814, 0x12})
/Users/rsc/go/src/runtime/panic.go:59 +0xa4 fp=0x4002855b00 sp=0x4002855ae0 pc=0x7ff786fedb24
runtime.panicBounds64(0x7ff787005480, 0x4002855b78)
/Users/rsc/go/src/runtime/panic.go:218 +0x70 fp=0x4002855b60 sp=0x4002855b00 pc=0x7ff786fedc20
runtime.panicBounds()
/Users/rsc/go/src/runtime/asm_arm64.s:1591 +0x38 fp=0x4002855c00 sp=0x4002855b60 pc=0x7ff78702c748
runtime.sellock({0x4002855dc8, 0x0, 0x4009dc03f0?}, {0x4002855dc4, 0x1, 0x2?})
/Users/rsc/go/src/runtime/select.go:37 +0xa0 fp=0x4002855c30 sp=0x4002855c00 pc=0x7ff787005480
runtime.selectgo(0x4002855dc8, 0x4000025dc0, 0x4000025db8?, 0x0, 0x100010000?, 0x1)
/Users/rsc/go/src/runtime/select.go:354 +0x864 fp=0x4002855d80 sp=0x4002855c30 pc=0x7ff787005ea4
golang.org/x/net/quic.(*Conn).waitOnDone(0x4000e1ca80?, {0x7ff78745ff30, 0x7ff787744460}, 0x40000feaf0)
/Users/rsc/src/golang.org/x/net/quic/conn.go:465 +0xe8 fp=0x4002855df0 sp=0x4002855d80 pc=0x7ff7872321c8
golang.org/x/net/quic.(*Stream).Close(0x4000e1ca80)
/Users/rsc/src/golang.org/x/net/quic/stream.go:503 +0xa4 fp=0x4002855e90 sp=0x4002855df0 pc=0x7ff7872543b4
golang.org/x/net/quic.testStreamTransfer(0x4000c3a540)
/Users/rsc/src/golang.org/x/net/quic/endpoint_test.go:73 +0x278 fp=0x4002855f60 sp=0x4002855e90 pc=0x7ff787273048
testing.tRunner(0x4000c3a540, 0x7ff7873ddda0)
/Users/rsc/go/src/testing/testing.go:1934 +0xac fp=0x4002855fb0 sp=0x4002855f60 pc=0x7ff7870c453c
testing.(*T).Run.gowrap1()
/Users/rsc/go/src/testing/testing.go:1997 +0x20 fp=0x4002855fd0 sp=0x4002855fb0 pc=0x7ff7870c5380
runtime.goexit({})
/Users/rsc/go/src/runtime/asm_arm64.s:1268 +0x4 fp=0x4002855fd0 sp=0x4002855fd0 pc=0x7ff78702c414
created by testing.(*T).Run in goroutine 7
/Users/rsc/go/src/testing/testing.go:1997 +0x358
...
On amd64:
% GOOS=windows GOARCH=amd64 go test -run=TestStreamTransfer
fatal error: sync: unlock of unlocked mutex
goroutine 1108 [running]:
internal/sync.fatal({0x7ff7df8dcffc?, 0xd36613e610a450fb?})
/Users/rsc/go/src/runtime/panic.go:1189 +0x18
internal/sync.(*Mutex).unlockSlow(0xc00000003c, 0xbf)
/Users/rsc/go/src/internal/sync/mutex.go:204 +0x35
internal/sync.(*Mutex).Unlock(...)
/Users/rsc/go/src/internal/sync/mutex.go:198
sync.(*Mutex).Unlock(...)
/Users/rsc/go/src/sync/mutex.go:65
crypto/tls.(*Conn).quicWaitForSignal(0xc000000010)
/Users/rsc/go/src/crypto/tls/quic.go:482 +0x4f
crypto/tls.(*Conn).quicReadHandshakeBytes(0xc000000010, 0x4)
/Users/rsc/go/src/crypto/tls/quic.go:380 +0x3c
crypto/tls.(*Conn).readHandshakeBytes(0x7ff7df4a5c5a?, 0x7ff7df7b7825?)
/Users/rsc/go/src/crypto/tls/conn.go:1082 +0x27
crypto/tls.(*Conn).readHandshake(0xc000000010, {0x22a7ac90c28, 0xc0011e4480})
/Users/rsc/go/src/crypto/tls/conn.go:1096 +0x39
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0xc000445c90)
/Users/rsc/go/src/crypto/tls/handshake_client_tls13.go:631 +0x12d
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc000445c90)
/Users/rsc/go/src/crypto/tls/handshake_client_tls13.go:137 +0x76c
crypto/tls.(*Conn).clientHandshake(0xc00113a008, {0x7ff7df97d520, 0xc00113e000})
/Users/rsc/go/src/crypto/tls/handshake_client.go:367 +0x828
crypto/tls.(*Conn).handshakeContext(0xc00113a008, {0x7ff7df97d408, 0x7ff7dfc78f40})
/Users/rsc/go/src/crypto/tls/conn.go:1575 +0x372
crypto/tls.(*Conn).HandshakeContext(...)
/Users/rsc/go/src/crypto/tls/conn.go:1515
created by crypto/tls.(*QUICConn).Start in goroutine 156
/Users/rsc/go/src/crypto/tls/quic.go:211 +0xba
and
% GOOS=windows GOARCH=amd64 go test -run=TestStreamTransfer
fatal error: index out of range
goroutine 752 gp=0xc000b61c00 m=6 mp=0xc000080808 [running]:
runtime.throw({0x7ff7df8d3329?, 0x1?})
/Users/rsc/go/src/runtime/panic.go:1227 +0x4d fp=0xc0015b1b20 sp=0xc0015b1af0 pc=0x7ff7df4a8a6d
runtime.panicCheck1(0x7ff7dfc3a960?, {0x7ff7df8d3329, 0x12})
/Users/rsc/go/src/runtime/panic.go:59 +0x94 fp=0xc0015b1b40 sp=0xc0015b1b20 pc=0x7ff7df470274
runtime.panicBounds64(0x7ff7df4872d0, 0xc0015b1bb0)
/Users/rsc/go/src/runtime/panic.go:218 +0x7e fp=0xc0015b1ba0 sp=0xc0015b1b40 pc=0x7ff7df47037e
runtime.panicBounds()
/Users/rsc/go/src/runtime/asm_amd64.s:2051 +0x68 fp=0xc0015b1c40 sp=0xc0015b1ba0 pc=0x7ff7df4b0428
runtime.sellock({0xc0015b1de0, 0x0, 0x9007ff7df43de40?}, {0xc0015b1ddc, 0x1, 0x10?})
/Users/rsc/go/src/runtime/select.go:37 +0x90 fp=0xc0015b1c68 sp=0xc0015b1c40 pc=0x7ff7df4872d0
runtime.selectgo(0xc0015b1de0, 0xc001503dd8, 0x142ebfccb30?, 0x0, 0x7ff7df7425e0?, 0x1)
/Users/rsc/go/src/runtime/select.go:354 +0xab9 fp=0xc0015b1da8 sp=0xc0015b1c68 pc=0x7ff7df487f59
golang.org/x/net/quic.(*Conn).waitOnDone(0xc001802380?, {0x7ff7df97d2f0, 0x7ff7dfc78f40}, 0xc0011c2310)
/Users/rsc/src/golang.org/x/net/quic/conn.go:465 +0xf7 fp=0xc0015b1e10 sp=0xc0015b1da8 pc=0x7ff7df719a37
golang.org/x/net/quic.(*Stream).Close(0xc001802380)
/Users/rsc/src/golang.org/x/net/quic/stream.go:503 +0x99 fp=0xc0015b1ea8 sp=0xc0015b1e10 pc=0x7ff7df742179
golang.org/x/net/quic.testStreamTransfer(0xc000b61a40)
/Users/rsc/src/golang.org/x/net/quic/endpoint_test.go:73 +0x2f6 fp=0xc0015b1f70 sp=0xc0015b1ea8 pc=0x7ff7df767976
testing.tRunner(0xc000b61a40, 0x7ff7df8fa990)
/Users/rsc/go/src/testing/testing.go:1934 +0xc3 fp=0xc0015b1fc0 sp=0xc0015b1f70 pc=0x7ff7df553783
testing.(*T).Run.gowrap1()
/Users/rsc/go/src/testing/testing.go:1997 +0x1b fp=0xc0015b1fe0 sp=0xc0015b1fc0 pc=0x7ff7df55485b
runtime.goexit({})
/Users/rsc/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0015b1fe8 sp=0xc0015b1fe0 pc=0x7ff7df4affc1
created by testing.(*T).Run in goroutine 7
/Users/rsc/go/src/testing/testing.go:1997 +0x44b
Comment From: rsc
Git bisect says this started at
commit 098426447182a216991923d5bf95cdd3ae836e2c
Author: qmuntal <quimmuntal@gmail.com>
AuthorDate: Wed Jul 2 14:55:04 2025 +0200
Commit: Quim Muntal <quimmuntal@gmail.com>
CommitDate: Mon Jul 28 11:52:21 2025 -0700
internal/poll: remove msg field from Windows' poll.operation
There is no need to keep the msg field in the poll.operation struct.
This skims down the size of os.File by 112 bytes.
Change-Id: I5c7b1f3989f9bb5f1748df2cba8128d9c479b35d
Reviewed-on: https://go-review.googlesource.com/c/go/+/685418
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Mark Freeman <mark@golang.org>
src/internal/poll/fd_windows.go | 94 +++++++++++++++++++----------------------
1 file changed, 43 insertions(+), 51 deletions(-)
Comment From: dmitshur
CC @qmuntal, @golang/windows.
Comment From: gopherbot
Change https://go.dev/cl/698437 mentions this issue: Revert "internal/poll: remove msg field from Windows' poll.operation"