Go version
go1.24.2 linux/amd64 and linux/arm64
Output of go env in your module/workspace:
n/a
What did you do?
Compiled programs with PGO enabled.
What did you see happen?
The programs makes no observable progress. We see gaps in metrics, can't connect to HTTP endpoints, no work getting done. We eventually have to kill the programs.
I have some example goroutine dumps after killing the affected programs with SIGQUIT. I don't have a good way to sanitize them, so I've opted to manually include the goroutines which seem "interesting" (basically, not parked). I can provide more information as needed. Here are some examples:
Example 1
SIGQUIT: quit
PC=0x4fc321 m=0 sigcode=0
goroutine 0 gp=0x5a60560 m=0 mp=0x5a62540 [idle]:
runtime.futex(0x5a62680, 0x80, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7ffc033100b0 sp=0x7ffc033100a8 pc=0x4fc321
runtime.futexsleep(0x0?, 0x5a62540?, 0x0?)
/usr/local/go/src/runtime/os_linux.go:75 +0x30 fp=0x7ffc03310100 sp=0x7ffc033100b0 pc=0x4ac390
runtime.notesleep(0x5a62680)
/usr/local/go/src/runtime/lock_futex.go:47 +0x87 fp=0x7ffc03310138 sp=0x7ffc03310100 pc=0x47bc67
runtime.mPark(...)
/usr/local/go/src/runtime/proc.go:1887
runtime.stopm()
/usr/local/go/src/runtime/proc.go:2907 +0x8c fp=0x7ffc03310168 sp=0x7ffc03310138 pc=0x4b824c
runtime.findRunnable()
/usr/local/go/src/runtime/proc.go:3644 +0xede fp=0x7ffc03310300 sp=0x7ffc03310168 pc=0x4ba09e
runtime.schedule(...)
/usr/local/go/src/runtime/proc.go:4017
runtime.park_m(0xc000899a40)
/usr/local/go/src/runtime/proc.go:4141 +0x295 fp=0x7ffc033103b0 sp=0x7ffc03310300 pc=0x4bb955
runtime.mcall()
/usr/local/go/src/runtime/asm_amd64.s:459 +0x4e fp=0x7ffc033103c8 sp=0x7ffc033103b0 pc=0x4f84ce
[ ... elided ... ]
goroutine 39308 gp=0xc008ab8a80 m=nil [semacquire]:
runtime.gopark(0x454bb95?, 0x7600000078e2?, 0x40?, 0x42?, 0x561ff48?)
/usr/local/go/src/runtime/proc.go:435 +0xce fp=0xc004469370 sp=0xc004469350 pc=0x4f098e
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:441
runtime.semacquire1(0x5a61298, 0x0, 0x0, 0x0, 0x12)
/usr/local/go/src/runtime/sema.go:188 +0x21d fp=0xc0044693d8 sp=0xc004469370 pc=0x4c88bd
runtime.semacquire(...)
/usr/local/go/src/runtime/sema.go:139
runtime.gcStart({0x76f35c56cf30?, 0x22?, 0x5a61980?})
/usr/local/go/src/runtime/mgc.go:668 +0x19a fp=0xc0044694d0 sp=0xc0044693d8 pc=0x48823a
runtime.mallocgcSmallScanNoHeader(...)
/usr/local/go/src/runtime/malloc.go:1425
runtime.mallocgc(0xe8, 0x3059ea0, 0x1)
/usr/local/go/src/runtime/malloc.go:1058 +0x40f fp=0xc004469578 sp=0xc0044694d0 pc=0x4ed04f
runtime.newobject(0xc012116a00?)
/usr/local/go/src/runtime/malloc.go:1714 +0x25 fp=0xc0044695a0 sp=0xc004469578 pc=0x47e825
runtime.CallersFrames(...)
/usr/local/go/src/runtime/symtab.go:81
runtime.makeTraceFrames(0xa, {0xc004106000, 0x42, 0x80})
/usr/local/go/src/runtime/tracestack.go:208 +0x65 fp=0xc0044696c0 sp=0xc0044695a0 pc=0x4e3de5
runtime.dumpStacksRec(0x0?, {{0x0?, 0x8?}, 0x80?, 0x1?}, {0xc004106000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:165 +0xe5 fp=0xc004469768 sp=0xc0044696c0 pc=0x4e3805
runtime.dumpStacksRec(0x0?, {{0x0?, 0x1?}, 0x80?, 0x6?}, {0xc004106000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc004469810 sp=0xc004469768 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0x6?}, 0x80?, 0x0?}, {0xc004106000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0044698b8 sp=0xc004469810 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0x6?}, 0x80?, 0x0?}, {0xc004106000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc004469960 sp=0xc0044698b8 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0xc008ad6a48?}, 0x29?, 0xc008ad6a30?}, {0xc004106000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc004469a08 sp=0xc004469960 pc=0x4e3b45
runtime.(*traceStackTable).dump(0x5a61820, 0xa)
/usr/local/go/src/runtime/tracestack.go:153 +0x69 fp=0xc004469a58 sp=0xc004469a08 pc=0x4e36a9
runtime.traceAdvance(0x1)
/usr/local/go/src/runtime/trace.go:589 +0x5db fp=0xc004469bb8 sp=0xc004469a58 pc=0x4f6b7b
runtime.StopTrace(...)
/usr/local/go/src/runtime/trace.go:318
runtime/trace.Stop()
/usr/local/go/src/runtime/trace/trace.go:148 +0x85 fp=0xc004469bf0 sp=0xc004469bb8 pc=0xa5f365
github.com/DataDog/dd-trace-go/v2/profiler.init.func4(0xc0041e2460)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profile.go:204 +0x1a5 fp=0xc004469c90 sp=0xc004469bf0 pc=0x22e5d25
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).runProfile(0xc0041e2460, 0x7)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profile.go:348 +0x290 fp=0xc004469e78 sp=0xc004469c90 pc=0x22e99f0
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect.func2(0x7)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:361 +0xe5 fp=0xc004469fc8 sp=0xc004469e78 pc=0x22ed305
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect.gowrap2()
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:373 +0x24 fp=0xc004469fe0 sp=0xc004469fc8 pc=0x22ed1e4
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc004469fe8 sp=0xc004469fe0 pc=0x4fa501
created by github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect in goroutine 81
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:356 +0x7c9
[ ... elided ... ]
rax 0xca
rbx 0x0
rcx 0x4fc323
rdx 0x0
rdi 0x5a62680
rsi 0x80
rbp 0x7ffc033100f0
rsp 0x7ffc033100a8
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x980ad
r13 0x1
r14 0x5a60560
r15 0x4
rip 0x4fc321
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Example 2
SIGQUIT: quit
PC=0x4fc321 m=0 sigcode=0
goroutine 0 gp=0x5a60560 m=0 mp=0x5a62540 [idle]:
runtime.futex(0x5a62680, 0x80, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fff0a80eb40 sp=0x7fff0a80eb38 pc=0x4fc321
runtime.futexsleep(0x7fff0a80ebb8?, 0x5a62540?, 0x7fff0a80ebd8?)
/usr/local/go/src/runtime/os_linux.go:75 +0x30 fp=0x7fff0a80eb90 sp=0x7fff0a80eb40 pc=0x4ac390
runtime.notesleep(0x5a62680)
/usr/local/go/src/runtime/lock_futex.go:47 +0x87 fp=0x7fff0a80ebc8 sp=0x7fff0a80eb90 pc=0x47bc67
runtime.mPark(...)
/usr/local/go/src/runtime/proc.go:1887
runtime.stopm()
/usr/local/go/src/runtime/proc.go:2907 +0x8c fp=0x7fff0a80ebf8 sp=0x7fff0a80ebc8 pc=0x4b824c
runtime.gcstopm()
/usr/local/go/src/runtime/proc.go:3218 +0xcf fp=0x7fff0a80ec20 sp=0x7fff0a80ebf8 pc=0x4b8d4f
runtime.findRunnable()
/usr/local/go/src/runtime/proc.go:3281 +0x1ca fp=0x7fff0a80edb8 sp=0x7fff0a80ec20 pc=0x4b938a
runtime.schedule(...)
/usr/local/go/src/runtime/proc.go:4017
runtime.goschedImpl(0xc00bd67a40, 0x78?)
/usr/local/go/src/runtime/proc.go:4175 +0x251 fp=0x7fff0a80ee20 sp=0x7fff0a80edb8 pc=0x4bc151
runtime.gopreempt_m(...)
/usr/local/go/src/runtime/proc.go:4192
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1074 +0x3d1 fp=0x7fff0a80ef58 sp=0x7fff0a80ee20 pc=0x4ce811
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:621 +0x7a fp=0x7fff0a80ef60 sp=0x7fff0a80ef58 pc=0x4f865a
goroutine 9508 gp=0xc0038fcc40 m=9 mp=0xc000850808 [trace reader (blocked)]:
goroutine running on other thread; stack unavailable
created by runtime/trace.Start in goroutine 9489
/usr/local/go/src/runtime/trace/trace.go:128 +0xcf
goroutine 9489 gp=0xc00beb7180 m=nil [runnable]:
runtime.funcfile({0x4cdf7b8?, 0x57f8180?}, 0x37?)
/usr/local/go/src/runtime/symtab.go:1132 +0x92 fp=0xc0054bf2c8 sp=0xc0054bf2c0 pc=0x4d3772
runtime.funcline1({0x43456da?, 0x43456da?}, 0x4c7616, 0x0)
/usr/local/go/src/runtime/symtab.go:1165 +0x689 fp=0xc0054bf3d0 sp=0xc0054bf2c8 pc=0x4f5949
runtime.(*Frames).Next(0xc0137e64b0)
/usr/local/go/src/runtime/symtab.go:197 +0x729 fp=0xc0054bf4f8 sp=0xc0054bf3d0 pc=0x4d2429
runtime.makeTraceFrames(0x5, {0xc015362000, 0x3, 0x80})
/usr/local/go/src/runtime/tracestack.go:210 +0x125 fp=0xc0054bf618 sp=0xc0054bf4f8 pc=0x4e3ea5
runtime.dumpStacksRec(0x0?, {{0x0?, 0x3d?}, 0x80?, 0xa?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:165 +0xe5 fp=0xc0054bf6c0 sp=0xc0054bf618 pc=0x4e3805
runtime.dumpStacksRec(0x0?, {{0x0?, 0xa?}, 0x80?, 0x4?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0054bf768 sp=0xc0054bf6c0 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0x4?}, 0x80?, 0x6?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0054bf810 sp=0xc0054bf768 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0x6?}, 0x80?, 0x0?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0054bf8b8 sp=0xc0054bf810 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0x6?}, 0x80?, 0x0?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0054bf960 sp=0xc0054bf8b8 pc=0x4e3b45
runtime.dumpStacksRec(0x0?, {{0x0?, 0xc000d0aa48?}, 0x29?, 0xc000d0aa30?}, {0xc015362000, 0x80, 0x80})
/usr/local/go/src/runtime/tracestack.go:199 +0x425 fp=0xc0054bfa08 sp=0xc0054bf960 pc=0x4e3b45
runtime.(*traceStackTable).dump(0x5a618d0, 0x5)
/usr/local/go/src/runtime/tracestack.go:153 +0x69 fp=0xc0054bfa58 sp=0xc0054bfa08 pc=0x4e36a9
runtime.traceAdvance(0x1)
/usr/local/go/src/runtime/trace.go:589 +0x5db fp=0xc0054bfbb8 sp=0xc0054bfa58 pc=0x4f6b7b
runtime.StopTrace(...)
/usr/local/go/src/runtime/trace.go:318
runtime/trace.Stop()
/usr/local/go/src/runtime/trace/trace.go:148 +0x85 fp=0xc0054bfbf0 sp=0xc0054bfbb8 pc=0xa5f365
github.com/DataDog/dd-trace-go/v2/profiler.init.func4(0xc005e94140)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profile.go:204 +0x1a5 fp=0xc0054bfc90 sp=0xc0054bfbf0 pc=0x22e5d25
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).runProfile(0xc005e94140, 0x7)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profile.go:348 +0x290 fp=0xc0054bfe78 sp=0xc0054bfc90 pc=0x22e99f0
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect.func2(0x7)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:361 +0xe5 fp=0xc0054bffc8 sp=0xc0054bfe78 pc=0x22ed305
github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect.gowrap2()
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:373 +0x24 fp=0xc0054bffe0 sp=0xc0054bffc8 pc=0x22ed1e4
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0054bffe8 sp=0xc0054bffe0 pc=0x4fa501
created by github.com/DataDog/dd-trace-go/v2/profiler.(*profiler).collect in goroutine 144
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.0.0-rc.23/profiler/profiler.go:356 +0x7c9
goroutine 9495 gp=0xc00bd67a40 m=nil [runnable]:
runtime/pprof.(*profMap).lookup(0xc0047ae040, {0xc01d1bd4d8, 0x2, 0x1e565?}, 0x0)
/usr/local/go/src/runtime/pprof/map.go:28 +0x4dd fp=0xc003749ed8 sp=0xc003749ed0 pc=0xa4539d
runtime/pprof.(*profileBuilder).addCPUData(0x5f5e100?, {0xc01d1bc748?, 0x7b8120?, 0xc00c200150?}, {0xc006c10590?, 0x39a41f8?, 0xc00c2009a0?})
/usr/local/go/src/runtime/pprof/proto.go:338 +0x105 fp=0xc003749f58 sp=0xc003749ed8 pc=0xa4d145
runtime/pprof.profileWriter({0x395c2a0?, 0xc0252e37d0?})
/usr/local/go/src/runtime/pprof/pprof.go:873 +0x70 fp=0xc003749fc0 sp=0xc003749f58 pc=0xa49910
runtime/pprof.StartCPUProfile.gowrap2()
/usr/local/go/src/runtime/pprof/pprof.go:856 +0x25 fp=0xc003749fe0 sp=0xc003749fc0 pc=0xa49805
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003749fe8 sp=0xc003749fe0 pc=0x4fa501
created by runtime/pprof.StartCPUProfile in goroutine 9483
/usr/local/go/src/runtime/pprof/pprof.go:856 +0x136
rax 0xca
rbx 0x0
rcx 0x4fc323
rdx 0x0
rdi 0x5a62680
rsi 0x80
rbp 0x7fff0a80eb80
rsp 0x7fff0a80eb38
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x7fff0a80ebd8
r13 0x1
r14 0x5a60560
r15 0x18
rip 0x4fc321
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Example 3 (from delve, analyzing a core file collected when we saw the issue
Note: goroutine 48 is running the execution tracer, and the leaf frame is calling morestack, waiting to be scheduled.(dlv) goroutines -with running -t
Goroutine 131 - User: src/runtime/sys_linux_arm64.s:138 runtime.usleep (0x2abc644) (thread 17) [preempted]
0 0x0000000002abc644 in runtime.usleep
at src/runtime/sys_linux_arm64.s:138
1 0x0000000002a8dcb8 in runtime.sighandler
at GOROOT/src/runtime/signal_unix.go:826
2 0x0000000002a8d098 in runtime.sigtrampgo
at GOROOT/src/runtime/signal_unix.go:490
3 0x0000000002a7d6a4 in internal/runtime/atomic.(*Uint32).CompareAndSwap
at GOROOT/src/internal/runtime/atomic/types.go:236
4 0x0000000002a7d6a4 in runtime.casgstatus
at GOROOT/src/runtime/proc.go:1211
5 0x0000000002a7d6a4 in runtime.park_m
at GOROOT/src/runtime/proc.go:4112
6 0x0000000002ab9314 in runtime.mcall
at src/runtime/asm_arm64.s:193
7 0x0000000002a9afa8 in runtime.gopark
at GOROOT/src/runtime/proc.go:435
8 0x0000000002a9afa8 in runtime.ReadTrace
at GOROOT/src/runtime/trace.go:758
9 0x0000000002f91bec in runtime/trace.Start.func1
at GOROOT/src/runtime/trace/trace.go:130
10 0x0000000002abb854 in runtime.goexit
at src/runtime/asm_arm64.s:1223
Goroutine 133 - User: src/runtime/sys_linux_arm64.s:138 runtime.usleep (0x2abc644) (thread 22)
0 0x0000000002abc644 in runtime.usleep
at src/runtime/sys_linux_arm64.s:138
1 0x0000000002a8dcb8 in runtime.sighandler
at GOROOT/src/runtime/signal_unix.go:826
2 0x0000000002a8d098 in runtime.sigtrampgo
at GOROOT/src/runtime/signal_unix.go:490
3 0x0000000002abcccc in runtime.futex
at src/runtime/sys_linux_arm64.s:650
4 0x0000000002a6d5fc in runtime.futexsleep
at GOROOT/src/runtime/os_linux.go:75
5 0x0000000002a3fd90 in runtime.notetsleep_internal
at GOROOT/src/runtime/lock_futex.go:70
6 0x0000000002a3fe70 in runtime.notetsleepg
at GOROOT/src/runtime/lock_futex.go:124
7 0x0000000002ab5570 in os/signal.signal_recv
at GOROOT/src/runtime/sigqueue.go:152
8 0x000000000336a6cc in os/signal.loop
at GOROOT/src/os/signal/signal_unix.go:23
9 0x0000000002abb854 in runtime.goexit
at src/runtime/asm_arm64.s:1223
Goroutine 107362 - User: src/runtime/sys_linux_arm64.s:138 runtime.usleep (0x2abc644) (thread 18)
0 0x0000000002abc644 in runtime.usleep
at src/runtime/sys_linux_arm64.s:138
1 0x0000000002a8dcb8 in runtime.sighandler
at GOROOT/src/runtime/signal_unix.go:826
2 0x0000000002a8d098 in runtime.sigtrampgo
at GOROOT/src/runtime/signal_unix.go:490
3 0x0000000002abccd0 in runtime.futex
at src/runtime/sys_linux_arm64.s:651
4 0x0000000002a6d65c in runtime.futexsleep
at GOROOT/src/runtime/os_linux.go:81
5 0x0000000002a3fcf4 in runtime.notetsleep_internal
at GOROOT/src/runtime/lock_futex.go:89
6 0x0000000002a3fdf4 in runtime.notetsleep
at GOROOT/src/runtime/lock_futex.go:112
7 0x0000000002a77918 in runtime.stopTheWorldWithSema
at GOROOT/src/runtime/proc.go:1612
8 0x0000000002a4c9d8 in runtime.gcStart.func2
at GOROOT/src/runtime/mgc.go:726
9 0x0000000002ab93ac in runtime.systemstack
at src/runtime/asm_arm64.s:244
10 0x0000000002ab9328 in runtime.systemstack_switch
at src/runtime/asm_arm64.s:201
(truncated)
[3 goroutines]
(dlv) goroutine 48 bt
0 0x0000000002a95ca0 in runtime.(*Func).Entry
at GOROOT/src/runtime/symtab.go:794
1 0x0000000002a94f6c in runtime.(*Frames).Next
at GOROOT/src/runtime/symtab.go:120
2 0x0000000002aa56a0 in runtime.makeTraceFrames
at GOROOT/src/runtime/tracestack.go:210
3 0x0000000002aa5154 in runtime.dumpStacksRec
at GOROOT/src/runtime/tracestack.go:165
4 0x0000000002aa5428 in runtime.dumpStacksRec
at GOROOT/src/runtime/tracestack.go:199
5 0x0000000002aa5428 in runtime.dumpStacksRec
at GOROOT/src/runtime/tracestack.go:199
6 0x0000000002aa5428 in runtime.dumpStacksRec
at GOROOT/src/runtime/tracestack.go:199
7 0x0000000002aa5428 in runtime.dumpStacksRec
at GOROOT/src/runtime/tracestack.go:199
8 0x0000000002aa5038 in runtime.(*traceStackTable).dump
at GOROOT/src/runtime/tracestack.go:153
9 0x0000000002ab7d1c in runtime.traceAdvance
at GOROOT/src/runtime/trace.go:589
10 0x0000000002f91d88 in runtime.StopTrace
at GOROOT/src/runtime/trace.go:318
11 0x0000000002f91d88 in runtime/trace.Stop
at GOROOT/src/runtime/trace/trace.go:148
12 0x0000000003b74790 in gopkg.in/DataDog/dd-trace-go.v1/profiler.init.func4
at external/in_gopkg_datadog_dd_trace_go_v1/profiler/profile.go:204
13 0x0000000003b77aac in gopkg.in/DataDog/dd-trace-go.v1/profiler.(*profiler).runProfile
at external/in_gopkg_datadog_dd_trace_go_v1/profiler/profile.go:348
14 0x0000000003b7ac28 in gopkg.in/DataDog/dd-trace-go.v1/profiler.(*profiler).collect.func2
at external/in_gopkg_datadog_dd_trace_go_v1/profiler/profiler.go:355
15 0x0000000003b7ab30 in gopkg.in/DataDog/dd-trace-go.v1/profiler.(*profiler).collect.gowrap2
at external/in_gopkg_datadog_dd_trace_go_v1/profiler/profiler.go:367
16 0x0000000002abb854 in runtime.goexit
at src/runtime/asm_arm64.s:1223
hot-node enabled increased budget=2000 for func=sync/atomic.(*Pointer[go.shape.chan struct {}]).Load
hot-node enabled increased budget=2000 for func=sync/atomic.(*Pointer[go.shape.struct { sync.poolDequeue; sync.next sync/atomic.Pointer[sync.poolChainElt]; sync.prev sync/atomic.Pointer[sy
nc.poolChainElt] }]).Load
hot-node enabled increased budget=2000 for func=internal/sync.(*HashTrieMap[go.shape.interface {},go.shape.interface {}]).init
hot-node enabled increased budget=2000 for func=internal/sync.(*entry[go.shape.interface {},go.shape.interface {}]).lookup
hot-node enabled increased budget=2000 for func=internal/sync.(*HashTrieMap[go.shape.interface {},go.shape.interface {}]).Load
hot-node enabled increased budget=2000 for func=INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorAscending
hot-budget check allows inlining for call errors.Is (cost 179) at INTERNAL/internal/tree/tree.go:129:15 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorAscending
hot-node enabled increased budget=2000 for func=INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorDescending
hot-budget check allows inlining for call errors.Is (cost 179) at INTERNAL/internal/tree/tree.go:113:15 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorDescending
hot-node enabled increased budget=2000 for func=INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).Add
hot-budget check allows inlining for call sync.(*RWMutex).Lock (cost 167) at INTERNAL/internal/tree/tree.go:61:14 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).Add
hot-node enabled increased budget=2000 for func=INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).DimensionChildren
hot-node enabled increased budget=2000 for func=INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).Children
hot-budget check allows inlining for call errors.Is (cost 179) at INTERNAL/internal/tree/tree.go:129:15 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorAscending
hot-budget check allows inlining for call errors.Is (cost 179) at INTERNAL/internal/tree/tree.go:113:15 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).PathVisitorDescending
hot-budget check allows inlining for call sync.(*RWMutex).Lock (cost 167) at INTERNAL/internal/tree/tree.go:61:14 in function INTERNAL/internal/tree.(*Multi[go.shape.*uint8]).Add
Not sure if it's relevant, but I've seen a goroutine running runtime.(*traceStackTable).dump via runtime.StopTrace in the stack dump from every instance I've seen of this issue.
What did you expect to see?
The programs function like normal. If we build the same source code but with PGO disabled, the programs work.
We have seen this for at least two programs at Datadog. We haven't reproduced the behavior in our development environments yet, but it does happen fairly consistently in production, so we can probably collect more diagnostics. I already have a core dump from one instance of this, and goroutine traceback dumps from others.
Comment From: gabyhelp
Related Issues
- runtime: apparent test deadlock on ppc64le #32327 (closed)
- maybe a deadlock in golang : runtime.selectgo while runtime.stopTheWorldWithSema #44047 (closed)
- runtime: apparent test deadlock on linux-amd64-race #32689 (closed)
- runtime/pprof: apparent deadlock in TestGoroutineSwitch on linux-armv6l #47505 (closed)
- runtime: deadlock during CPU profiling #8407 (closed)
- cgo: GOMAXPROCS deadlock #2001 (closed)
- runtime: TestPanicSystemstack is flaky on linux #33626 (closed)
- runtime: TestCgoCCodeSIGPROF hangs #14766 (closed)
- runtime: broken work conservation due to CL 310850 #45867 (closed)
- cmd/go: `go test` deadlocked without enforcing timeouts when killed with ^C #60203 (closed)
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
Comment From: cherrymui
Thanks for the report. Perhaps you can try building with -gcflags=runtime=-d=pgodebug=1 to see what PGO does to the runtime with your profile.
If the failure is reliably reproducible, perhaps you can also try using the -d=pgohash=NNN flag to bisect to particular points of inlining/devirtualization.
Comment From: nsrip-dd
Thanks for the report. Perhaps you can try building with -gcflags=runtime=-d=pgodebug=1 to see what PGO does to the runtime with your profile.
Here's the output of GOOS=linux GOARCH=arm64 go1.24.2 build -gcflags=runtime=-d=pgodebug=1 -pgo=default.pgo runtime using the profile from "example 3" above:
Runtime PGO debug output
# runtime
hot-callsite-thres-from-CDF=0.00248954391555467
hot-node enabled increased budget=2000 for func=runtime.acquirem
hot-node enabled increased budget=2000 for func=runtime.readgstatus
hot-node enabled increased budget=2000 for func=runtime.cheaprand
hot-node enabled increased budget=2000 for func=runtime.nanotime
hot-node enabled increased budget=2000 for func=runtime.cheaprandn
hot-node enabled increased budget=2000 for func=runtime.findmoduledatap
hot-node enabled increased budget=2000 for func=runtime.add
hot-node enabled increased budget=2000 for func=runtime.findfunc
hot-node enabled increased budget=2000 for func=runtime.findnull
hot-node enabled increased budget=2000 for func=runtime.gostringnocopy
hot-budget check allows inlining for call runtime.findnull (cost 123) at ../../../../../../sdk/go1.24.2/src/runtime/string.go:601:60 in function runtime.gostringnocopy
hot-budget check allows inlining for call runtime.gostringnocopy (cost 142) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:733:23 in function runtime.(*moduledata).funcName
hot-node enabled increased budget=2000 for func=runtime.releasem
hot-node enabled increased budget=2000 for func=runtime.readvarint
hot-node enabled increased budget=2000 for func=runtime.step
hot-node enabled increased budget=2000 for func=runtime.traceEnabled
hot-node enabled increased budget=2000 for func=runtime.traceAcquireEnabled
hot-node enabled increased budget=2000 for func=runtime.traceAcquire
hot-budget check allows inlining for call runtime.traceAcquireEnabled (cost 137) at ../../../../../../sdk/go1.24.2/src/runtime/traceruntime.go:190:28 in function runtime.traceAcquire
hot-node enabled increased budget=2000 for func=runtime.traceClockNow
hot-node enabled increased budget=2000 for func=runtime.(*guintptr).cas
hot-node enabled increased budget=2000 for func=runtime.(*mSpanStateBox).get
hot-node enabled increased budget=2000 for func=runtime.(*mspan).base
hot-node enabled increased budget=2000 for func=runtime.addb
hot-node enabled increased budget=2000 for func=runtime.(*gcBits).bytep
hot-node enabled increased budget=2000 for func=runtime.(*gcBits).bitp
hot-node enabled increased budget=2000 for func=runtime.(*mspan).markBitsForIndex
hot-node enabled increased budget=2000 for func=runtime.markBits.isMarked
hot-node enabled increased budget=2000 for func=runtime.arenaIndex
hot-node enabled increased budget=2000 for func=runtime.spanHasNoSpecials
hot-node enabled increased budget=2000 for func=runtime.(*mspan).countAlloc
hot-node enabled increased budget=2000 for func=runtime.(*gcBitsArena).tryAlloc
hot-node enabled increased budget=2000 for func=runtime.spanOf
hot-node enabled increased budget=2000 for func=runtime.(*mspan).divideByElemSize
hot-node enabled increased budget=2000 for func=runtime.(*mspan).objIndex
hot-node enabled increased budget=2000 for func=runtime.markBits.setMarked
hot-node enabled increased budget=2000 for func=runtime.pageIndexOf
hot-node enabled increased budget=2000 for func=runtime.(*gcWork).putFast
hot-node enabled increased budget=2000 for func=runtime.(*lfstack).pop
hot-node enabled increased budget=2000 for func=runtime.(*pageAlloc).chunkOf
hot-node enabled increased budget=2000 for func=runtime.(*pallocBits).summarize
hot-node enabled increased budget=2000 for func=runtime.mergeSummaries
hot-node enabled increased budget=2000 for func=runtime.(*pageAlloc).update
hot-budget check allows inlining for call runtime.(*pallocBits).summarize (cost 315) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:497:31 in function runtime.(*pageAlloc).update
hot-budget check allows inlining for call runtime.mergeSummaries (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:552:25 in function runtime.(*pageAlloc).update
hot-node enabled increased budget=2000 for func=runtime.(*mSpanStateBox).set
hot-node enabled increased budget=2000 for func=runtime.(*mspan).init
hot-node enabled increased budget=2000 for func=runtime.heapBitsInSpan
hot-node enabled increased budget=2000 for func=runtime.(*mspan).refillAllocCache
hot-node enabled increased budget=2000 for func=runtime.headTailIndex.head
hot-node enabled increased budget=2000 for func=runtime.(*atomicSpanSetSpinePointer).Load
hot-node enabled increased budget=2000 for func=runtime.(*mcentral).fullSwept
hot-node enabled increased budget=2000 for func=runtime.(*mheap).freeMSpanLocked
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).memoryLimitHeapGoal
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).heapGoalInternal
hot-budget check allows inlining for call runtime.(*gcControllerState).memoryLimitHeapGoal (cost 157) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:925:37 in function runtime.(*gcControllerState).heapGoalInternal
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).revise
hot-node enabled increased budget=2000 for func=runtime.spanOfUnchecked
hot-node enabled increased budget=2000 for func=runtime.rand
hot-budget check allows inlining for call internal/chacha8rand.(*State).Refill (cost 131) at ../../../../../../sdk/go1.24.2/src/runtime/rand.go:177:11 in function runtime.rand
hot-node enabled increased budget=2000 for func=runtime.(*timeHistogram).record
hot-node enabled increased budget=2000 for func=runtime.isSystemGoroutine
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:1341:15 in function runtime.isSystemGoroutine
hot-node enabled increased budget=2000 for func=internal/runtime/atomic.(*Pointer[go.shape.a0c91c71fd368b5d30f8a04d1e4f14a4186fd3423a1957aa58b1e03c3b3735dd]).CompareAndSwapNoWB
hot-node enabled increased budget=2000 for func=runtime.(*gQueue).pop
hot-node enabled increased budget=2000 for func=runtime.runqget
hot-node enabled increased budget=2000 for func=runtime.(*gList).pop
hot-node enabled increased budget=2000 for func=runtime.runqgrab
hot-node enabled increased budget=2000 for func=runtime.funcdata
hot-node enabled increased budget=2000 for func=runtime.eqslice
hot-node enabled increased budget=2000 for func=runtime.lock
hot-node enabled increased budget=2000 for func=runtime.lockWithRank
hot-node enabled increased budget=2000 for func=runtime.lock2
hot-node enabled increased budget=2000 for func=runtime.unlock
hot-node enabled increased budget=2000 for func=runtime.unlockWithRank
hot-node enabled increased budget=2000 for func=runtime.unlock2
hot-node enabled increased budget=2000 for func=runtime.(*mLockProfile).recordUnlock
hot-node enabled increased budget=2000 for func=runtime.(*unwinder).initAt
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:200:15 in function runtime.(*unwinder).initAt
hot-node enabled increased budget=2000 for func=runtime.funcInfo.entry
hot-node enabled increased budget=2000 for func=runtime.(*moduledata).textAddr
hot-node enabled increased budget=2000 for func=runtime.(*unwinder).resolveInternal
hot-node enabled increased budget=2000 for func=runtime.(*unwinder).finishInternal
hot-node enabled increased budget=2000 for func=runtime.funcspdelta
hot-node enabled increased budget=2000 for func=runtime.pcvalue
hot-budget check allows inlining for call runtime.step (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1045:15 in function runtime.pcvalue
hot-node enabled increased budget=2000 for func=runtime.tracebackPCs
hot-node enabled increased budget=2000 for func=runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.(*unwinder).finishInternal (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:447:19 in function runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:450:17 in function runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.(*unwinder).resolveInternal (cost 758) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:513:19 in function runtime.(*unwinder).next
hot-node enabled increased budget=2000 for func=runtime.cgocall
hot-node enabled increased budget=2000 for func=runtime.reentersyscall
hot-node enabled increased budget=2000 for func=runtime.casgstatus
hot-budget check allows inlining for call runtime.(*timeHistogram).record (cost 97) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1295:25 in function runtime.casgstatus
hot-node enabled increased budget=2000 for func=runtime.goready
hot-node enabled increased budget=2000 for func=runtime.ready
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1051:23 in function runtime.ready
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1052:12 in function runtime.ready
hot-node enabled increased budget=2000 for func=runtime.traceWriter.event
hot-node enabled increased budget=2000 for func=runtime.(*sysMemStat).add
hot-budget check allows inlining for call runtime.traceWriter.event (cost 318) at ../../../../../../sdk/go1.24.2/src/runtime/traceevent.go:133:21 in function runtime.traceEventWriter.event
hot-node enabled increased budget=2000 for func=runtime.callers
hot-budget check allows inlining for call runtime.tracebackPCs (cost 490) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:1100:19 in function runtime.callers.func1
hot-node enabled increased budget=2000 for func=runtime.(*unwinder).init
hot-budget check allows inlining for call runtime.(*unwinder).initAt (cost 639) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:129:10 in function runtime.(*unwinder).init
hot-node enabled increased budget=2000 for func=runtime.runqput
hot-node enabled increased budget=2000 for func=runtime.wakep
hot-node enabled increased budget=2000 for func=runtime.pidleget
hot-node enabled increased budget=2000 for func=runtime.(*consistentHeapStats).acquire
hot-node enabled increased budget=2000 for func=runtime.(*consistentHeapStats).release
hot-node enabled increased budget=2000 for func=runtime.(*mcentral).uncacheSpan
hot-node enabled increased budget=2000 for func=runtime.(*sweepLocked).sweep
hot-node enabled increased budget=2000 for func=runtime.(*bucket).mp
hot-node enabled increased budget=2000 for func=runtime.newMarkBits
hot-node enabled increased budget=2000 for func=runtime.atomicwb
hot-node enabled increased budget=2000 for func=runtime.(*wbBuf).get2
hot-node enabled increased budget=2000 for func=runtime.wbBufFlush1
hot-node enabled increased budget=2000 for func=runtime.findObject
hot-node enabled increased budget=2000 for func=runtime.greyobject
hot-node enabled increased budget=2000 for func=runtime.getempty
hot-node enabled increased budget=2000 for func=runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1354:32 in function runtime.(*mheap).allocSpan
hot-node enabled increased budget=2000 for func=runtime.(*pageAlloc).allocToCache
hot-budget check allows inlining for call runtime.(*pageAlloc).update (cost 1018) at ../../../../../../sdk/go1.24.2/src/runtime/mpagecache.go:168:10 in function runtime.(*pageAlloc).allocToCache
hot-node enabled increased budget=2000 for func=runtime.(*scavengeIndex).alloc
hot-node enabled increased budget=2000 for func=runtime.sysUsed
hot-node enabled increased budget=2000 for func=runtime.(*fixalloc).alloc
hot-node enabled increased budget=2000 for func=runtime.(*mheap).allocMSpanLocked
hot-budget check allows inlining for call runtime.(*fixalloc).alloc (cost 331) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1125:53 in function runtime.(*mheap).allocMSpanLocked
hot-node enabled increased budget=2000 for func=runtime.(*pageAlloc).scavengeOne
hot-node enabled increased budget=2000 for func=runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.(*mspan).init (cost 83) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1393:8 in function runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.newMarkBits (cost 643) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1426:29 in function runtime.(*mheap).initSpan
hot-node enabled increased budget=2000 for func=runtime.(*mheap).allocNeedsZero
hot-node enabled increased budget=2000 for func=runtime.newAllocBits
hot-budget check allows inlining for call runtime.newMarkBits (cost 643) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:2519:20 in function runtime.newAllocBits
hot-node enabled increased budget=2000 for func=runtime.(*lfstack).push
hot-node enabled increased budget=2000 for func=runtime.trygetfull
hot-node enabled increased budget=2000 for func=runtime.(*spanSet).push
hot-node enabled increased budget=2000 for func=runtime.(*atomicHeadTailIndex).incTail
hot-node enabled increased budget=2000 for func=runtime.(*spanSetBlockAlloc).alloc
hot-node enabled increased budget=2000 for func=runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1648:27 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1650:29 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1653:37 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1664:28 in function runtime.(*mheap).freeSpanLocked
hot-node enabled increased budget=2000 for func=runtime.(*pageAlloc).free
hot-budget check allows inlining for call runtime.(*pageAlloc).update (cost 1018) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:969:10 in function runtime.(*pageAlloc).free
hot-node enabled increased budget=2000 for func=runtime.(*scavengeIndex).free
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).update
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:874:24 in function runtime.(*gcControllerState).update
hot-budget check allows inlining for call runtime.(*gcControllerState).revise (cost 338) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:890:11 in function runtime.(*gcControllerState).update
hot-node enabled increased budget=2000 for func=runtime.stackpoolfree
hot-node enabled increased budget=2000 for func=runtime.stackfree
hot-node enabled increased budget=2000 for func=runtime.stackcacherelease
hot-budget check allows inlining for call runtime.stackpoolfree (cost 347) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:302:16 in function runtime.stackcacherelease
hot-node enabled increased budget=2000 for func=runtime.stackalloc
hot-node enabled increased budget=2000 for func=runtime.stackpoolalloc
hot-node enabled increased budget=2000 for func=runtime.exitsyscall
hot-node enabled increased budget=2000 for func=runtime.exitsyscallfast
hot-node enabled increased budget=2000 for func=runtime.schedule
hot-node enabled increased budget=2000 for func=runtime.findRunnable
hot-node enabled increased budget=2000 for func=runtime.(*timers).adjust
hot-node enabled increased budget=2000 for func=runtime.(*timers).run
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).findRunnableGCWorker
hot-budget check allows inlining for call runtime.(*lfstack).push (cost 136) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:812:27 in function runtime.(*gcControllerState).findRunnableGCWorker
hot-node enabled increased budget=2000 for func=runtime.globrunqget
hot-node enabled increased budget=2000 for func=runtime.netpoll
hot-node enabled increased budget=2000 for func=runtime.stealWork
hot-budget check allows inlining for call runtime.runqgrab (cost 192) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:6988:15 in function runtime.runqsteal
hot-node enabled increased budget=2000 for func=runtime.newInlineUnwinder
hot-node enabled increased budget=2000 for func=runtime.pcdatavalue1
hot-budget check allows inlining for call runtime.pcvalue (cost 1228) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1215:17 in function runtime.pcdatavalue1
hot-node enabled increased budget=2000 for func=runtime.(*inlineUnwinder).next
hot-node enabled increased budget=2000 for func=runtime.stkbucket
hot-node enabled increased budget=2000 for func=runtime.pcdatavalue
hot-budget check allows inlining for call runtime.pcvalue (cost 1228) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1207:17 in function runtime.pcdatavalue
hot-node enabled increased budget=2000 for func=runtime.park_m
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4112:12 in function runtime.park_m
hot-budget check allows inlining for call runtime.schedule (cost 1236) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4141:10 in function runtime.park_m
hot-node enabled increased budget=2000 for func=runtime.gopark
hot-node enabled increased budget=2000 for func=runtime.(*waitq).dequeue
hot-node enabled increased budget=2000 for func=runtime.resolveNameOff
hot-node enabled increased budget=2000 for func=runtime.getGCMask
hot-node enabled increased budget=2000 for func=runtime.(*wbBuf).get1
hot-node enabled increased budget=2000 for func=runtime.readUintptr
hot-node enabled increased budget=2000 for func=runtime.(*mspan).typePointersOfType
hot-node enabled increased budget=2000 for func=runtime.(*mspan).heapBitsSmallForAddr
hot-node enabled increased budget=2000 for func=runtime.(*mspan).typePointersOfUnchecked
hot-budget check allows inlining for call runtime.(*mspan).heapBitsSmallForAddr (cost 156) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:184:78 in function runtime.(*mspan).typePointersOfUnchecked
hot-node enabled increased budget=2000 for func=runtime.typePointers.nextFast
hot-node enabled increased budget=2000 for func=runtime.typePointers.next
hot-node enabled increased budget=2000 for func=runtime.bulkBarrierPreWrite
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfType (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:460:28 in function runtime.bulkBarrierPreWrite
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:477:25 in function runtime.bulkBarrierPreWrite
hot-node enabled increased budget=2000 for func=runtime.typedmemmove
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:158:22 in function runtime.typedmemmove
hot-node enabled increased budget=2000 for func=runtime.acquireSudog
hot-node enabled increased budget=2000 for func=runtime.(*waitq).enqueue
hot-node enabled increased budget=2000 for func=runtime.chanparkcommit
hot-node enabled increased budget=2000 for func=runtime.releaseSudog
hot-node enabled increased budget=2000 for func=runtime.chansend
hot-budget check allows inlining for call runtime.typedmemmove (cost 1205) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:242:15 in function runtime.chansend
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:259:22 in function runtime.chansend
hot-node enabled increased budget=2000 for func=runtime.typedmemclr
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:366:22 in function runtime.typedmemclr
hot-node enabled increased budget=2000 for func=runtime.recv
hot-budget check allows inlining for call runtime.typedmemmove (cost 1205) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:727:15 in function runtime.recv
hot-node enabled increased budget=2000 for func=runtime.(*timers).cleanHead
hot-node enabled increased budget=2000 for func=runtime.(*timers).addHeap
hot-node enabled increased budget=2000 for func=runtime.chanrecv
hot-budget check allows inlining for call runtime.recv (cost 1617) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:604:8 in function runtime.chanrecv
hot-node enabled increased budget=2000 for func=runtime.interhash
hot-node enabled increased budget=2000 for func=runtime.typehash
hot-node enabled increased budget=2000 for func=runtime.nilinterhash
hot-node enabled increased budget=2000 for func=runtime.strequal
hot-node enabled increased budget=2000 for func=runtime.ifaceeq
hot-node enabled increased budget=2000 for func=runtime.efaceeq
hot-node enabled increased budget=2000 for func=runtime.(*activeSweep).begin
hot-node enabled increased budget=2000 for func=runtime.(*sweepLocker).tryAcquire
hot-node enabled increased budget=2000 for func=runtime.(*activeSweep).end
hot-node enabled increased budget=2000 for func=runtime.(*itabTableType).find
hot-node enabled increased budget=2000 for func=runtime.resolveTypeOff
hot-node enabled increased budget=2000 for func=runtime.rtype.typeOff
hot-budget check allows inlining for call runtime.resolveTypeOff (cost 388) at ../../../../../../sdk/go1.24.2/src/runtime/type.go:363:23 in function runtime.rtype.typeOff
hot-node enabled increased budget=2000 for func=runtime.(*gcWork).tryGetFast
hot-node enabled increased budget=2000 for func=runtime.(*gcWork).tryGet
hot-budget check allows inlining for call runtime.trygetfull (cost 118) at ../../../../../../sdk/go1.24.2/src/runtime/mgcwork.go:216:21 in function runtime.(*gcWork).tryGet
hot-node enabled increased budget=2000 for func=runtime.scanblock
hot-budget check allows inlining for call runtime.findObject (cost 288) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1362:42 in function runtime.scanblock
hot-budget check allows inlining for call runtime.greyobject (cost 651) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1363:17 in function runtime.scanblock
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:285:11 in function runtime.markrootBlock
hot-node enabled increased budget=2000 for func=runtime.scanobject
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfUnchecked (cost 321) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1429:33 in function runtime.scanobject
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1436:25 in function runtime.scanobject
hot-budget check allows inlining for call runtime.findObject (cost 288) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1462:40 in function runtime.scanobject
hot-budget check allows inlining for call runtime.greyobject (cost 651) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1463:15 in function runtime.scanobject
hot-node enabled increased budget=2000 for func=runtime.markrootSpans
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).addScannableStack
hot-node enabled increased budget=2000 for func=runtime.adjustpointer
hot-node enabled increased budget=2000 for func=runtime.adjustctxt
hot-node enabled increased budget=2000 for func=runtime.(*stkframe).argMapInternal
hot-node enabled increased budget=2000 for func=runtime.(*stkframe).getStackMap
hot-budget check allows inlining for call runtime.pcdatavalue (cost 1270) at ../../../../../../sdk/go1.24.2/src/runtime/stkframe.go:172:23 in function runtime.(*stkframe).getStackMap
hot-budget check allows inlining for call runtime.(*stkframe).argMapInternal (cost 668) at ../../../../../../sdk/go1.24.2/src/runtime/stkframe.go:215:40 in function runtime.(*stkframe).getStackMap
hot-node enabled increased budget=2000 for func=runtime.adjustpointers
hot-node enabled increased budget=2000 for func=runtime.(*stackObjectRecord).gcdata
hot-node enabled increased budget=2000 for func=runtime.adjustframe
hot-budget check allows inlining for call runtime.adjustpointers (cost 306) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:697:17 in function runtime.adjustframe
hot-budget check allows inlining for call runtime.adjustpointers (cost 306) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:705:17 in function runtime.adjustframe
hot-node enabled increased budget=2000 for func=runtime.copystack
hot-budget check allows inlining for call runtime.stackalloc (cost 1280) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:874:19 in function runtime.copystack
hot-budget check allows inlining for call runtime.adjustctxt (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:919:12 in function runtime.copystack
hot-node enabled increased budget=2000 for func=runtime.shrinkstack
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:1209:15 in function runtime.shrinkstack
hot-node enabled increased budget=2000 for func=runtime.(*stackScanState).addObject
hot-budget check allows inlining for call runtime.getempty (cost 927) at ../../../../../../sdk/go1.24.2/src/runtime/mgcstack.go:274:48 in function runtime.(*stackScanState).addObject
hot-node enabled increased budget=2000 for func=runtime.scanframeworker
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1055:12 in function runtime.scanframeworker
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1060:12 in function runtime.scanframeworker
hot-node enabled increased budget=2000 for func=runtime.(*stackScanState).getPtr
hot-node enabled increased budget=2000 for func=runtime.scanstack
hot-budget check allows inlining for call runtime.shrinkstack (cost 729) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:878:14 in function runtime.scanstack
hot-budget check allows inlining for call runtime.(*stackScanState).getPtr (cost 207) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:942:34 in function runtime.scanstack
hot-node enabled increased budget=2000 for func=runtime.gcFlushBgCredit
hot-node enabled increased budget=2000 for func=runtime.markroot
hot-budget check allows inlining for call runtime.markrootSpans (cost 752) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:193:16 in function runtime.markroot
hot-node enabled increased budget=2000 for func=runtime.gcDrainN
hot-node enabled increased budget=2000 for func=runtime.(*semaRoot).queue
hot-node enabled increased budget=2000 for func=runtime.goparkunlock
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:441:8 in function runtime.goparkunlock
hot-node enabled increased budget=2000 for func=runtime.semacquire1
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:159:19 in function runtime.semacquire1
hot-budget check allows inlining for call runtime.(*semaRoot).queue (cost 399) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:187:13 in function runtime.semacquire1
hot-budget check allows inlining for call runtime.goparkunlock (cost 228) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:188:15 in function runtime.semacquire1
hot-node enabled increased budget=2000 for func=runtime.(*semaRoot).dequeue
hot-node enabled increased budget=2000 for func=runtime.semrelease1
hot-budget check allows inlining for call runtime.(*semaRoot).dequeue (cost 427) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:222:33 in function runtime.semrelease1
hot-node enabled increased budget=2000 for func=runtime.pollFractionalWorkerExit
hot-node enabled increased budget=2000 for func=runtime.gcDrain
hot-budget check allows inlining for call runtime.markroot (cost 1520) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1186:12 in function runtime.gcDrain
hot-budget check allows inlining for call runtime.pollFractionalWorkerExit (cost 113) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1187:28 in function runtime.gcDrain
hot-node enabled increased budget=2000 for func=runtime.(*atomicHeadTailIndex).load
hot-node enabled increased budget=2000 for func=runtime.headTailIndex.split
hot-node enabled increased budget=2000 for func=runtime.(*atomicHeadTailIndex).cas
hot-node enabled increased budget=2000 for func=runtime.(*spanSet).pop
hot-node enabled increased budget=2000 for func=runtime.(*sweepClass).update
hot-node enabled increased budget=2000 for func=runtime.(*mheap).nextSpanForSweep
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:103:29 in function runtime.(*mheap).nextSpanForSweep
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:105:32 in function runtime.(*mheap).nextSpanForSweep
hot-node enabled increased budget=2000 for func=runtime.sweepone
hot-budget check allows inlining for call runtime.(*mheap).nextSpanForSweep (cost 876) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:370:31 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:385:28 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*activeSweep).end (cost 206) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:402:18 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*bucket).mp (cost 84) at ../../../../../../sdk/go1.24.2/src/runtime/mprof.go:406:13 in function runtime.mProf_FlushLocked
hot-node enabled increased budget=2000 for func=runtime.(*gcControllerState).trigger
hot-budget check allows inlining for call runtime.(*gcControllerState).heapGoalInternal (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:1109:40 in function runtime.(*gcControllerState).trigger
hot-node enabled increased budget=2000 for func=runtime.bool2int
hot-node enabled increased budget=2000 for func=runtime.makeSpanClass
hot-node enabled increased budget=2000 for func=runtime.gcAssistAlloc
hot-node enabled increased budget=2000 for func=runtime.deductAssistCredit
hot-node enabled increased budget=2000 for func=runtime.getMCache
hot-node enabled increased budget=2000 for func=runtime.nextFreeFast
hot-node enabled increased budget=2000 for func=runtime.(*mspan).nextFreeIndex
hot-budget check allows inlining for call runtime.(*mspan).refillAllocCache (cost 98) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:1105:21 in function runtime.(*mspan).nextFreeIndex
hot-node enabled increased budget=2000 for func=runtime.deductSweepCredit
hot-budget check allows inlining for call runtime.sweepone (cost 1624) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:935:14 in function runtime.deductSweepCredit
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:945:22 in function runtime.deductSweepCredit
hot-node enabled increased budget=2000 for func=runtime.(*mheap).reclaim
hot-node enabled increased budget=2000 for func=runtime.(*mheap).alloc
hot-budget check allows inlining for call runtime.(*mheap).reclaim (cost 754) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:968:13 in function runtime.(*mheap).alloc.func1
hot-budget check allows inlining for call runtime.(*mheap).alloc (cost 81) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:254:19 in function runtime.(*mcentral).grow
hot-node enabled increased budget=2000 for func=runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:113:31 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:121:32 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:125:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:141:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:145:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*activeSweep).end (cost 206) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:152:22 in function runtime.(*mcentral).cacheSpan
hot-node enabled increased budget=2000 for func=runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*mcentral).uncacheSpan (cost 321) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:160:43 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:163:38 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:172:29 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*gcControllerState).update (cost 657) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:213:21 in function runtime.(*mcache).refill
hot-node enabled increased budget=2000 for func=runtime.(*mcache).nextFree
hot-budget check allows inlining for call runtime.(*mspan).nextFreeIndex (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:957:30 in function runtime.(*mcache).nextFree
hot-budget check allows inlining for call runtime.(*mcache).refill (cost 1800) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:964:11 in function runtime.(*mcache).nextFree
hot-node enabled increased budget=2000 for func=runtime.gcmarknewobject
hot-node enabled increased budget=2000 for func=runtime.spanHasSpecials
hot-node enabled increased budget=2000 for func=runtime.addspecial
hot-budget check allows inlining for call runtime.(*fixalloc).alloc (cost 331) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:2307:57 in function runtime.setprofilebucket
hot-budget check allows inlining for call runtime.(*gcControllerState).trigger (cost 421) at ../../../../../../sdk/go1.24.2/src/runtime/mgc.go:608:37 in function runtime.gcTrigger.test
hot-node enabled increased budget=2000 for func=runtime.gcDrainMarkWorkerFractional
hot-node enabled increased budget=2000 for func=runtime.gcDrainMarkWorkerIdle
hot-node enabled increased budget=2000 for func=runtime.(*spanSet).reset
hot-node enabled increased budget=2000 for func=runtime.mallocgcTiny
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1211:18 in function runtime.mallocgcTiny
hot-node enabled increased budget=2000 for func=runtime.mallocgcSmallNoscan
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1313:18 in function runtime.mallocgcSmallNoscan
hot-node enabled increased budget=2000 for func=runtime.(*mspan).writeHeapBitsSmall
hot-node enabled increased budget=2000 for func=runtime.heapSetTypeNoHeader
hot-budget check allows inlining for call runtime.(*mspan).writeHeapBitsSmall (cost 296) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:725:37 in function runtime.heapSetTypeNoHeader
hot-node enabled increased budget=2000 for func=runtime.mallocgcSmallScanNoHeader
hot-budget check allows inlining for call runtime.heapSetTypeNoHeader (cost 307) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1377:37 in function runtime.mallocgcSmallScanNoHeader
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1404:18 in function runtime.mallocgcSmallScanNoHeader
hot-node enabled increased budget=2000 for func=runtime.mallocgcSmallScanHeader
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1497:18 in function runtime.mallocgcSmallScanHeader
hot-node enabled increased budget=2000 for func=runtime.mallocgc
hot-budget check allows inlining for call runtime.deductAssistCredit (cost 87) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1044:21 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcTiny (cost 947) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1053:31 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcSmallNoscan (cost 918) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1055:38 in function runtime.mallocgc
hot-node enabled increased budget=2000 for func=runtime.newobject
hot-node enabled increased budget=2000 for func=runtime.rawstring
hot-node enabled increased budget=2000 for func=runtime.bulkBarrierPreWriteSrcOnly
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfType (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:521:28 in function runtime.bulkBarrierPreWriteSrcOnly
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:527:24 in function runtime.bulkBarrierPreWriteSrcOnly
hot-node enabled increased budget=2000 for func=runtime.makeslicecopy
hot-node enabled increased budget=2000 for func=runtime.makechan
hot-node enabled increased budget=2000 for func=runtime.chansend1
hot-node enabled increased budget=2000 for func=runtime.closechan
hot-node enabled increased budget=2000 for func=runtime.selectnbsend
hot-node enabled increased budget=2000 for func=runtime.selectnbrecv
hot-node enabled increased budget=2000 for func=runtime.gfput
hot-budget check allows inlining for call runtime.stackfree (cost 1271) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5187:12 in function runtime.gfput
hot-node enabled increased budget=2000 for func=runtime.gdestroy
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4317:12 in function runtime.gdestroy
hot-budget check allows inlining for call runtime.isSystemGoroutine (cost 372) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4319:22 in function runtime.gdestroy
hot-budget check allows inlining for call runtime.gfput (cost 1732) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4361:7 in function runtime.gdestroy
hot-node enabled increased budget=2000 for func=runtime.gfget
hot-node enabled increased budget=2000 for func=runtime.gostartcallfn
hot-node enabled increased budget=2000 for func=runtime.saveAncestors
hot-node enabled increased budget=2000 for func=runtime.newproc1
hot-budget check allows inlining for call runtime.gfget (cost 496) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5042:15 in function runtime.newproc1
hot-budget check allows inlining for call runtime.gostartcallfn (cost 158) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5074:15 in function runtime.newproc1
hot-budget check allows inlining for call runtime.saveAncestors (cost 162) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5077:32 in function runtime.newproc1
hot-budget check allows inlining for call runtime.isSystemGoroutine (cost 372) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5079:22 in function runtime.newproc1
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5104:23 in function runtime.newproc1
hot-node enabled increased budget=2000 for func=runtime.(*timer).modify
hot-node enabled increased budget=2000 for func=runtime.convT64
hot-node enabled increased budget=2000 for func=runtime.convTstring
hot-node enabled increased budget=2000 for func=runtime.convTslice
hot-node enabled increased budget=2000 for func=runtime.mapiterinit
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:100:12 in function runtime.mapiterinit
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:101:12 in function runtime.mapiterinit
hot-node enabled increased budget=2000 for func=runtime.mapiternext
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:150:12 in function runtime.mapiternext
hot-node enabled increased budget=2000 for func=runtime.roundupsize
hot-node enabled increased budget=2000 for func=runtime.makemap_small
hot-node enabled increased budget=2000 for func=runtime.mapdelete
hot-budget check allows inlining for call internal/runtime/maps.(*Map).Delete (cost 1010) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:158:10 in function runtime.mapdelete
hot-node enabled increased budget=2000 for func=runtime.mapIterStart
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:171:9 in function runtime.mapIterStart
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:172:9 in function runtime.mapIterStart
hot-node enabled increased budget=2000 for func=runtime.mapIterNext
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:183:9 in function runtime.mapIterNext
hot-node enabled increased budget=2000 for func=runtime.mapclone2
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:341:11 in function runtime.mapclone2
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:342:15 in function runtime.mapclone2
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:342:47 in function runtime.mapclone2
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:183:21 in function runtime.wbZero
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:197:21 in function runtime.wbMove
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:322:22 in function runtime.typedslicecopy
hot-node enabled increased budget=2000 for func=runtime.netpollblockcommit
hot-node enabled increased budget=2000 for func=runtime.popDefer
hot-node enabled increased budget=2000 for func=runtime.(*_panic).nextFrame
hot-node enabled increased budget=2000 for func=runtime.(*_panic).start
hot-budget check allows inlining for call runtime.schedule (cost 1236) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4310:10 in function runtime.goexit0
hot-node enabled increased budget=2000 for func=runtime.gorecover
hot-node enabled increased budget=2000 for func=runtime.funcMaxSPDelta
hot-budget check allows inlining for call runtime.step (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1191:15 in function runtime.funcMaxSPDelta
hot-node enabled increased budget=2000 for func=runtime.newproc
hot-node enabled increased budget=2000 for func=runtime.newproc.func1
hot-budget check allows inlining for call runtime.runqput (cost 145) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5024:10 in function runtime.newproc.func1
hot-budget check allows inlining for call runtime.wakep (cost 503) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5027:9 in function runtime.newproc.func1
hot-node enabled increased budget=2000 for func=runtime.procPin
hot-node enabled increased budget=2000 for func=runtime.procUnpin
hot-node enabled increased budget=2000 for func=runtime.sellock
hot-node enabled increased budget=2000 for func=runtime.selunlock
hot-node enabled increased budget=2000 for func=runtime.selectgo
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:318:21 in function runtime.selectgo
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:351:8 in function runtime.selectgo
hot-budget check allows inlining for call runtime.sellock (cost 95) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:354:9 in function runtime.selectgo
hot-node enabled increased budget=2000 for func=runtime.makeslice
hot-node enabled increased budget=2000 for func=runtime.growslice
hot-budget check allows inlining for call runtime.bulkBarrierPreWriteSrcOnly (cost 690) at ../../../../../../sdk/go1.24.2/src/runtime/slice.go:280:30 in function runtime.growslice
hot-node enabled increased budget=2000 for func=runtime.concatstrings
hot-budget check allows inlining for call runtime.concatstrings (cost 245) at ../../../../../../sdk/go1.24.2/src/runtime/string.go:61:22 in function runtime.concatstring2
hot-node enabled increased budget=2000 for func=runtime.slicebytetostring
hot-node enabled increased budget=2000 for func=runtime.ReadTrace.func2
hot-budget check allows inlining for call runtime.findnull (cost 123) at ../../../../../../sdk/go1.24.2/src/runtime/string.go:601:60 in function runtime.gostringnocopy
hot-budget check allows inlining for call runtime.gostringnocopy (cost 142) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:733:23 in function runtime.(*moduledata).funcName
hot-budget check allows inlining for call runtime.traceAcquireEnabled (cost 137) at ../../../../../../sdk/go1.24.2/src/runtime/traceruntime.go:190:28 in function runtime.traceAcquire
hot-budget check allows inlining for call runtime.(*pallocBits).summarize (cost 315) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:497:31 in function runtime.(*pageAlloc).update
hot-budget check allows inlining for call runtime.mergeSummaries (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:552:25 in function runtime.(*pageAlloc).update
hot-budget check allows inlining for call runtime.(*gcControllerState).memoryLimitHeapGoal (cost 157) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:925:37 in function runtime.(*gcControllerState).heapGoalInternal
hot-budget check allows inlining for call internal/chacha8rand.(*State).Refill (cost 131) at ../../../../../../sdk/go1.24.2/src/runtime/rand.go:177:11 in function runtime.rand
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:1341:15 in function runtime.isSystemGoroutine
hot-budget check allows inlining for call runtime.lock2 (cost 799) at ../../../../../../sdk/go1.24.2/src/runtime/lockrank_off.go:24:7 in function runtime.lockWithRank
hot-budget check allows inlining for call runtime.unlock2 (cost 301) at ../../../../../../sdk/go1.24.2/src/runtime/lockrank_off.go:35:9 in function runtime.unlockWithRank
hot-budget check allows inlining for call runtime.(*mLockProfile).recordUnlock (cost 140) at ../../../../../../sdk/go1.24.2/src/runtime/lock_spinbit.go:271:32 in function runtime.unlock2
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:200:15 in function runtime.(*unwinder).initAt
hot-budget check allows inlining for call runtime.(*unwinder).resolveInternal (cost 758) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:224:19 in function runtime.(*unwinder).initAt
hot-budget check allows inlining for call runtime.(*moduledata).textAddr (cost 140) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:873:25 in function runtime.funcInfo.entry
hot-budget check allows inlining for call runtime.pcvalue (cost 1228) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1174:17 in function runtime.funcspdelta
hot-budget check allows inlining for call runtime.step (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1045:15 in function runtime.pcvalue
hot-budget check allows inlining for call runtime.(*unwinder).next (cost 1841) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:623:43 in function runtime.tracebackPCs
hot-budget check allows inlining for call runtime.newInlineUnwinder (cost 171) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:628:34 in function runtime.tracebackPCs
hot-budget check allows inlining for call runtime.(*inlineUnwinder).next (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:628:92 in function runtime.tracebackPCs
hot-budget check allows inlining for call runtime.(*unwinder).finishInternal (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:447:19 in function runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:450:17 in function runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.(*unwinder).resolveInternal (cost 758) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:513:19 in function runtime.(*unwinder).next
hot-budget check allows inlining for call runtime.reentersyscall (cost 1001) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4531:16 in function runtime.entersyscall
hot-budget check allows inlining for call runtime.(*timeHistogram).record (cost 97) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1295:25 in function runtime.casgstatus
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1051:23 in function runtime.ready
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1052:12 in function runtime.ready
hot-budget check allows inlining for call runtime.runqput (cost 145) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1057:9 in function runtime.ready
hot-budget check allows inlining for call runtime.wakep (cost 503) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:1058:7 in function runtime.ready
hot-budget check allows inlining for call runtime.traceWriter.event (cost 318) at ../../../../../../sdk/go1.24.2/src/runtime/traceevent.go:133:21 in function runtime.traceEventWriter.event
hot-budget check allows inlining for call runtime.tracebackPCs (cost 490) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:1100:19 in function runtime.callers.func1
hot-budget check allows inlining for call runtime.(*unwinder).initAt (cost 639) at ../../../../../../sdk/go1.24.2/src/runtime/traceback.go:129:10 in function runtime.(*unwinder).init
hot-budget check allows inlining for call runtime.pidleget (cost 222) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:6697:21 in function runtime.pidlegetSpinning
hot-budget check allows inlining for call runtime.(*spanSet).push (cost 667) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:244:24 in function runtime.(*mcentral).uncacheSpan
hot-budget check allows inlining for call runtime.newMarkBits (cost 643) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:682:28 in function runtime.(*sweepLocked).sweep
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:757:39 in function runtime.(*sweepLocked).sweep
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:759:30 in function runtime.(*sweepLocked).sweep
hot-budget check allows inlining for call runtime.(*spanSet).push (cost 667) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:779:61 in function runtime.(*sweepLocked).sweep
hot-budget check allows inlining for call runtime.wbBufFlush1 (cost 561) at ../../../../../../sdk/go1.24.2/src/runtime/mwbbuf.go:181:14 in function runtime.wbBufFlush.func1
hot-budget check allows inlining for call runtime.findObject (cost 288) at ../../../../../../sdk/go1.24.2/src/runtime/mwbbuf.go:240:36 in function runtime.wbBufFlush1
hot-budget check allows inlining for call runtime.(*pageAlloc).allocToCache (cost 1607) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1198:29 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*mheap).allocMSpanLocked (cost 464) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1266:25 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*mheap).initSpan (cost 1191) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1346:12 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1354:32 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1357:27 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1359:29 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1362:37 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1375:28 in function runtime.(*mheap).allocSpan
hot-budget check allows inlining for call runtime.(*pageAlloc).update (cost 1018) at ../../../../../../sdk/go1.24.2/src/runtime/mpagecache.go:168:10 in function runtime.(*pageAlloc).allocToCache
hot-budget check allows inlining for call runtime.(*scavengeIndex).alloc (cost 143) at ../../../../../../sdk/go1.24.2/src/runtime/mpagecache.go:171:20 in function runtime.(*pageAlloc).allocToCache
hot-budget check allows inlining for call runtime.(*fixalloc).alloc (cost 331) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1125:53 in function runtime.(*mheap).allocMSpanLocked
hot-budget check allows inlining for call runtime.(*pageAlloc).scavengeOne (cost 1303) at ../../../../../../sdk/go1.24.2/src/runtime/mgcscavenge.go:683:29 in function runtime.(*pageAlloc).scavenge.func1
hot-budget check allows inlining for call runtime.(*mspan).init (cost 83) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1393:8 in function runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.(*mheap).allocNeedsZero (cost 174) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1394:21 in function runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.newMarkBits (cost 643) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1426:29 in function runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.newAllocBits (cost 647) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1427:29 in function runtime.(*mheap).initSpan
hot-budget check allows inlining for call runtime.newMarkBits (cost 643) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:2519:20 in function runtime.newAllocBits
hot-budget check allows inlining for call runtime.(*atomicHeadTailIndex).incTail (cost 98) at ../../../../../../sdk/go1.24.2/src/runtime/mspanset.go:76:35 in function runtime.(*spanSet).push
hot-budget check allows inlining for call runtime.(*spanSetBlockAlloc).alloc (cost 122) at ../../../../../../sdk/go1.24.2/src/runtime/mspanset.go:125:33 in function runtime.(*spanSet).push
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1648:27 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*sysMemStat).add (cost 94) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1650:29 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1653:37 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1664:28 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*pageAlloc).free (cost 1839) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:1667:14 in function runtime.(*mheap).freeSpanLocked
hot-budget check allows inlining for call runtime.(*scavengeIndex).free (cost 262) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:947:20 in function runtime.(*pageAlloc).free
hot-budget check allows inlining for call runtime.(*pageAlloc).update (cost 1018) at ../../../../../../sdk/go1.24.2/src/runtime/mpagealloc.go:969:10 in function runtime.(*pageAlloc).free
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:874:24 in function runtime.(*gcControllerState).update
hot-budget check allows inlining for call runtime.(*gcControllerState).revise (cost 338) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:890:11 in function runtime.(*gcControllerState).update
hot-budget check allows inlining for call runtime.stackcacherelease (cost 545) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:496:22 in function runtime.stackfree
hot-budget check allows inlining for call runtime.stackpoolfree (cost 347) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:302:16 in function runtime.stackcacherelease
hot-budget check allows inlining for call runtime.stackpoolalloc (cost 545) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:282:22 in function runtime.stackcacherefill
hot-budget check allows inlining for call runtime.exitsyscallfast (cost 431) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4674:20 in function runtime.exitsyscall
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4706:13 in function runtime.exitsyscall
hot-budget check allows inlining for call runtime.schedule (cost 1236) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4175:10 in function runtime.goschedImpl
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3244:12 in function runtime.execute
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3258:23 in function runtime.execute
hot-budget check allows inlining for call runtime.(*gcControllerState).findRunnableGCWorker (cost 914) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3310:48 in function runtime.findRunnable
hot-budget check allows inlining for call runtime.globrunqget (cost 201) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3347:20 in function runtime.findRunnable
hot-budget check allows inlining for call runtime.netpoll (cost 605) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3362:28 in function runtime.findRunnable
hot-budget check allows inlining for call runtime.stealWork (cost 514) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:3386:49 in function runtime.findRunnable
hot-budget check allows inlining for call runtime.(*timers).run (cost 554) at ../../../../../../sdk/go1.24.2/src/runtime/time.go:985:19 in function runtime.(*timers).check
hot-budget check allows inlining for call runtime.(*timers).adjust (cost 570) at ../../../../../../sdk/go1.24.2/src/runtime/time.go:1001:13 in function runtime.(*timers).check
hot-budget check allows inlining for call runtime.(*lfstack).push (cost 136) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:812:27 in function runtime.(*gcControllerState).findRunnableGCWorker
hot-budget check allows inlining for call runtime.runqgrab (cost 192) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:6988:15 in function runtime.runqsteal
hot-budget check allows inlining for call runtime.pcdatavalue1 (cost 1270) at ../../../../../../sdk/go1.24.2/src/runtime/symtabinl.go:82:22 in function runtime.(*inlineUnwinder).resolveInternal
hot-budget check allows inlining for call runtime.pcvalue (cost 1228) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1215:17 in function runtime.pcdatavalue1
hot-budget check allows inlining for call runtime.pcvalue (cost 1228) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1207:17 in function runtime.pcdatavalue
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4112:12 in function runtime.park_m
hot-budget check allows inlining for call runtime.schedule (cost 1236) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4141:10 in function runtime.park_m
hot-budget check allows inlining for call runtime.(*mspan).heapBitsSmallForAddr (cost 156) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:184:78 in function runtime.(*mspan).typePointersOfUnchecked
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfType (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:460:28 in function runtime.bulkBarrierPreWrite
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:477:25 in function runtime.bulkBarrierPreWrite
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:158:22 in function runtime.typedmemmove
hot-budget check allows inlining for call runtime.typedmemmove (cost 1205) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:242:15 in function runtime.chansend
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:259:22 in function runtime.chansend
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:283:8 in function runtime.chansend
hot-budget check allows inlining for call runtime.releaseSudog (cost 738) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:302:14 in function runtime.chansend
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:366:22 in function runtime.typedmemclr
hot-budget check allows inlining for call runtime.typedmemmove (cost 1205) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:727:15 in function runtime.recv
hot-budget check allows inlining for call runtime.(*timers).cleanHead (cost 555) at ../../../../../../sdk/go1.24.2/src/runtime/time.go:682:14 in function runtime.(*timer).maybeAdd
hot-budget check allows inlining for call runtime.(*timers).addHeap (cost 258) at ../../../../../../sdk/go1.24.2/src/runtime/time.go:692:13 in function runtime.(*timer).maybeAdd
hot-budget check allows inlining for call runtime.recv (cost 1617) at ../../../../../../sdk/go1.24.2/src/runtime/chan.go:604:8 in function runtime.chanrecv
hot-budget check allows inlining for call runtime.typehash (cost 984) at ../../../../../../sdk/go1.24.2/src/runtime/alg.go:242:16 in function runtime.typehash
hot-budget check allows inlining for call runtime.typehash (cost 984) at ../../../../../../sdk/go1.24.2/src/runtime/alg.go:242:16 in function runtime.typehash
hot-budget check allows inlining for call runtime.resolveTypeOff (cost 388) at ../../../../../../sdk/go1.24.2/src/runtime/type.go:363:23 in function runtime.rtype.typeOff
hot-budget check allows inlining for call runtime.trygetfull (cost 118) at ../../../../../../sdk/go1.24.2/src/runtime/mgcwork.go:216:21 in function runtime.(*gcWork).tryGet
hot-budget check allows inlining for call runtime.findObject (cost 288) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1362:42 in function runtime.scanblock
hot-budget check allows inlining for call runtime.greyobject (cost 651) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1363:17 in function runtime.scanblock
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:285:11 in function runtime.markrootBlock
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfUnchecked (cost 321) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1429:33 in function runtime.scanobject
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1436:25 in function runtime.scanobject
hot-budget check allows inlining for call runtime.findObject (cost 288) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1462:40 in function runtime.scanobject
hot-budget check allows inlining for call runtime.greyobject (cost 651) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1463:15 in function runtime.scanobject
hot-budget check allows inlining for call runtime.pcdatavalue (cost 1270) at ../../../../../../sdk/go1.24.2/src/runtime/stkframe.go:172:23 in function runtime.(*stkframe).getStackMap
hot-budget check allows inlining for call runtime.(*stkframe).argMapInternal (cost 668) at ../../../../../../sdk/go1.24.2/src/runtime/stkframe.go:215:40 in function runtime.(*stkframe).getStackMap
hot-budget check allows inlining for call runtime.adjustpointers (cost 306) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:697:17 in function runtime.adjustframe
hot-budget check allows inlining for call runtime.adjustpointers (cost 306) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:705:17 in function runtime.adjustframe
hot-budget check allows inlining for call runtime.stackalloc (cost 1280) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:874:19 in function runtime.copystack
hot-budget check allows inlining for call runtime.adjustctxt (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:919:12 in function runtime.copystack
hot-budget check allows inlining for call runtime.(*unwinder).init (cost 646) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:934:12 in function runtime.copystack
hot-budget check allows inlining for call runtime.(*unwinder).next (cost 1841) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:934:38 in function runtime.copystack
hot-budget check allows inlining for call runtime.adjustframe (cost 957) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:935:14 in function runtime.copystack
hot-budget check allows inlining for call runtime.stackfree (cost 1271) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:942:11 in function runtime.copystack
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:1209:15 in function runtime.shrinkstack
hot-budget check allows inlining for call runtime.getempty (cost 927) at ../../../../../../sdk/go1.24.2/src/runtime/mgcstack.go:274:48 in function runtime.(*stackScanState).addObject
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1055:12 in function runtime.scanframeworker
hot-budget check allows inlining for call runtime.scanblock (cost 1126) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1060:12 in function runtime.scanframeworker
hot-budget check allows inlining for call runtime.(*stackScanState).addObject (cost 1182) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1083:19 in function runtime.scanframeworker
hot-budget check allows inlining for call runtime.shrinkstack (cost 729) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:878:14 in function runtime.scanstack
hot-budget check allows inlining for call runtime.(*stackScanState).getPtr (cost 207) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:942:34 in function runtime.scanstack
hot-budget check allows inlining for call runtime.markrootSpans (cost 752) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:193:16 in function runtime.markroot
hot-budget check allows inlining for call runtime.gcDrainN (cost 581) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:670:22 in function runtime.gcAssistAlloc1
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:441:8 in function runtime.goparkunlock
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:159:19 in function runtime.semacquire1
hot-budget check allows inlining for call runtime.(*semaRoot).queue (cost 399) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:187:13 in function runtime.semacquire1
hot-budget check allows inlining for call runtime.goparkunlock (cost 228) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:188:15 in function runtime.semacquire1
hot-budget check allows inlining for call runtime.(*semaRoot).dequeue (cost 427) at ../../../../../../sdk/go1.24.2/src/runtime/sema.go:222:33 in function runtime.semrelease1
hot-budget check allows inlining for call runtime.markroot (cost 1520) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1186:12 in function runtime.gcDrain
hot-budget check allows inlining for call runtime.pollFractionalWorkerExit (cost 113) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1187:28 in function runtime.gcDrain
hot-budget check allows inlining for call runtime.(*gcWork).tryGet (cost 307) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1215:18 in function runtime.gcDrain
hot-budget check allows inlining for call runtime.gcFlushBgCredit (cost 414) at ../../../../../../sdk/go1.24.2/src/runtime/mgcmark.go:1236:20 in function runtime.gcDrain
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:103:29 in function runtime.(*mheap).nextSpanForSweep
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:105:32 in function runtime.(*mheap).nextSpanForSweep
hot-budget check allows inlining for call runtime.(*mheap).nextSpanForSweep (cost 876) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:370:31 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:385:28 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*activeSweep).end (cost 206) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:402:18 in function runtime.sweepone
hot-budget check allows inlining for call runtime.(*bucket).mp (cost 84) at ../../../../../../sdk/go1.24.2/src/runtime/mprof.go:406:13 in function runtime.mProf_FlushLocked
hot-budget check allows inlining for call runtime.(*gcControllerState).heapGoalInternal (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/mgcpacer.go:1109:40 in function runtime.(*gcControllerState).trigger
hot-budget check allows inlining for call runtime.(*mspan).refillAllocCache (cost 98) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:1105:21 in function runtime.(*mspan).nextFreeIndex
hot-budget check allows inlining for call runtime.sweepone (cost 1624) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:935:14 in function runtime.deductSweepCredit
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:945:22 in function runtime.deductSweepCredit
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:902:31 in function runtime.(*mheap).reclaimChunk
hot-budget check allows inlining for call runtime.(*mheap).reclaim (cost 754) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:968:13 in function runtime.(*mheap).alloc.func1
hot-budget check allows inlining for call runtime.(*mheap).alloc (cost 81) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:254:19 in function runtime.(*mcentral).grow
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:113:31 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:121:32 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:125:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*spanSet).pop (cost 352) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:141:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*sweepLocker).tryAcquire (cost 101) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:145:29 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*activeSweep).end (cost 206) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:152:22 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*mspan).refillAllocCache (cost 98) at ../../../../../../sdk/go1.24.2/src/runtime/mcentral.go:191:20 in function runtime.(*mcentral).cacheSpan
hot-budget check allows inlining for call runtime.(*mcentral).uncacheSpan (cost 321) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:160:43 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*consistentHeapStats).acquire (cost 189) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:163:38 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*consistentHeapStats).release (cost 164) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:172:29 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*gcControllerState).update (cost 657) at ../../../../../../sdk/go1.24.2/src/runtime/mcache.go:213:21 in function runtime.(*mcache).refill
hot-budget check allows inlining for call runtime.(*mspan).nextFreeIndex (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:957:30 in function runtime.(*mcache).nextFree
hot-budget check allows inlining for call runtime.(*mcache).refill (cost 1800) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:964:11 in function runtime.(*mcache).nextFree
hot-budget check allows inlining for call runtime.(*mspan).nextFreeIndex (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:968:30 in function runtime.(*mcache).nextFree
hot-budget check allows inlining for call runtime.(*fixalloc).alloc (cost 331) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:2307:57 in function runtime.setprofilebucket
hot-budget check allows inlining for call runtime.addspecial (cost 546) at ../../../../../../sdk/go1.24.2/src/runtime/mheap.go:2311:16 in function runtime.setprofilebucket
hot-budget check allows inlining for call runtime.stkbucket (cost 1011) at ../../../../../../sdk/go1.24.2/src/runtime/mprof.go:450:16 in function runtime.mProf_Malloc
hot-budget check allows inlining for call runtime.(*gcControllerState).trigger (cost 421) at ../../../../../../sdk/go1.24.2/src/runtime/mgc.go:608:37 in function runtime.gcTrigger.test
hot-budget check allows inlining for call runtime.(*spanSet).reset (cost 419) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:256:26 in function runtime.finishsweep_m
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1211:18 in function runtime.mallocgcTiny
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1313:18 in function runtime.mallocgcSmallNoscan
hot-budget check allows inlining for call runtime.(*mspan).writeHeapBitsSmall (cost 296) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:725:37 in function runtime.heapSetTypeNoHeader
hot-budget check allows inlining for call runtime.heapSetTypeNoHeader (cost 307) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1377:37 in function runtime.mallocgcSmallScanNoHeader
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1404:18 in function runtime.mallocgcSmallScanNoHeader
hot-budget check allows inlining for call runtime.gcmarknewobject (cost 304) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1497:18 in function runtime.mallocgcSmallScanHeader
hot-budget check allows inlining for call runtime.deductAssistCredit (cost 87) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1044:21 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcTiny (cost 947) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1053:31 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcSmallNoscan (cost 918) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1055:38 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcSmallScanNoHeader (cost 1218) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1058:43 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.mallocgcSmallScanHeader (cost 955) at ../../../../../../sdk/go1.24.2/src/runtime/malloc.go:1060:41 in function runtime.mallocgc
hot-budget check allows inlining for call runtime.(*itabTableType).find (cost 82) at ../../../../../../sdk/go1.24.2/src/runtime/iface.go:65:15 in function runtime.getitab
hot-budget check allows inlining for call runtime.(*mspan).typePointersOfType (cost 110) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:521:28 in function runtime.bulkBarrierPreWriteSrcOnly
hot-budget check allows inlining for call runtime.typePointers.next (cost 230) at ../../../../../../sdk/go1.24.2/src/runtime/mbitmap.go:527:24 in function runtime.bulkBarrierPreWriteSrcOnly
hot-budget check allows inlining for call runtime.stackfree (cost 1271) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5187:12 in function runtime.gfput
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4317:12 in function runtime.gdestroy
hot-budget check allows inlining for call runtime.isSystemGoroutine (cost 372) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4319:22 in function runtime.gdestroy
hot-budget check allows inlining for call runtime.gfput (cost 1732) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4361:7 in function runtime.gdestroy
hot-budget check allows inlining for call runtime.gfget (cost 496) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5042:15 in function runtime.newproc1
hot-budget check allows inlining for call runtime.gostartcallfn (cost 158) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5074:15 in function runtime.newproc1
hot-budget check allows inlining for call runtime.saveAncestors (cost 162) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5077:32 in function runtime.newproc1
hot-budget check allows inlining for call runtime.isSystemGoroutine (cost 372) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5079:22 in function runtime.newproc1
hot-budget check allows inlining for call runtime.traceAcquire (cost 149) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5104:23 in function runtime.newproc1
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5119:12 in function runtime.newproc1
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:100:12 in function runtime.mapiterinit
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:101:12 in function runtime.mapiterinit
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/linkname_swiss.go:150:12 in function runtime.mapiternext
hot-budget check allows inlining for call internal/runtime/maps.(*Map).Delete (cost 1010) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:158:10 in function runtime.mapdelete
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:171:9 in function runtime.mapIterStart
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:172:9 in function runtime.mapIterStart
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:183:9 in function runtime.mapIterNext
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Init (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:341:11 in function runtime.mapclone2
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:342:15 in function runtime.mapclone2
hot-budget check allows inlining for call internal/runtime/maps.(*Iter).Next (cost 1334) at ../../../../../../sdk/go1.24.2/src/runtime/map_swiss.go:342:47 in function runtime.mapclone2
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:183:21 in function runtime.wbZero
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:197:21 in function runtime.wbMove
hot-budget check allows inlining for call runtime.bulkBarrierPreWrite (cost 1119) at ../../../../../../sdk/go1.24.2/src/runtime/mbarrier.go:322:22 in function runtime.typedslicecopy
hot-budget check allows inlining for call runtime.sweepone (cost 1624) at ../../../../../../sdk/go1.24.2/src/runtime/mgcsweep.go:297:15 in function runtime.bgsweep
hot-budget check allows inlining for call runtime.popDefer (cost 312) at ../../../../../../sdk/go1.24.2/src/runtime/panic.go:924:12 in function runtime.(*_panic).nextDefer
hot-budget check allows inlining for call runtime.(*_panic).nextFrame (cost 89) at ../../../../../../sdk/go1.24.2/src/runtime/panic.go:929:18 in function runtime.(*_panic).nextDefer
hot-budget check allows inlining for call runtime.schedule (cost 1236) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:4310:10 in function runtime.goexit0
hot-budget check allows inlining for call runtime.step (cost 182) at ../../../../../../sdk/go1.24.2/src/runtime/symtab.go:1191:15 in function runtime.funcMaxSPDelta
hot-budget check allows inlining for call runtime.runqput (cost 145) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5024:10 in function runtime.newproc.func1
hot-budget check allows inlining for call runtime.wakep (cost 503) at ../../../../../../sdk/go1.24.2/src/runtime/proc.go:5027:9 in function runtime.newproc.func1
hot-budget check allows inlining for call runtime.acquireSudog (cost 346) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:318:21 in function runtime.selectgo
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:351:8 in function runtime.selectgo
hot-budget check allows inlining for call runtime.sellock (cost 95) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:354:9 in function runtime.selectgo
hot-budget check allows inlining for call runtime.selunlock (cost 106) at ../../../../../../sdk/go1.24.2/src/runtime/select.go:443:11 in function runtime.selectgo
hot-budget check allows inlining for call runtime.bulkBarrierPreWriteSrcOnly (cost 690) at ../../../../../../sdk/go1.24.2/src/runtime/slice.go:280:30 in function runtime.growslice
hot-budget check allows inlining for call runtime.findfunc (cost 216) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:1084:18 in function runtime.newstack
hot-budget check allows inlining for call runtime.funcMaxSPDelta (cost 371) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:1085:32 in function runtime.newstack
hot-budget check allows inlining for call runtime.casgstatus (cost 1123) at ../../../../../../sdk/go1.24.2/src/runtime/stack.go:1112:12 in function runtime.newstack
hot-budget check allows inlining for call runtime.concatstrings (cost 245) at ../../../../../../sdk/go1.24.2/src/runtime/string.go:61:22 in function runtime.concatstring2
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/trace.go:758:9 in function runtime.ReadTrace
If the failure is reliably reproducible, perhaps you can also try using the -d=pgohash=NNN flag to bisect to particular points of inlining/devirtualization.
I'm not finding any references on this. How does it work? Do I set pgohash to different binary values as in https://research.swtch.com/bisect#use_case?
Comment From: cherrymui
I'm not finding any references on this. How does it work? Do I set pgohash to different binary values as in https://research.swtch.com/bisect#use_case?
Yeah, pretty much. https://research.swtch.com/bisect#try_bisect is the section more relevant. I think you can use https://pkg.go.dev/golang.org/x/tools/cmd/bisect and try bisect -compile=pgo <command to reproduce> (unverified). Thanks.
Comment From: rhysh
In example 3, I see a goroutine in runtime.stopTheWorldWithSema trying to preempt all other goroutines (https://github.com/golang/go/blob/go1.24.2/src/runtime/proc.go#L1612).
I also see the trace reader goroutine in runtime.casgstatus. "What are the chances" that we happened to catch it there — I take that to mean that it's in a tight loop between running and trying to sleep.
How is the trace reader goroutine supposed to yield during trace shutdown? CC @mknyszek @cagedmantis
ReadTrace is in a call to gopark (https://github.com/golang/go/blob/go1.24.2/src/runtime/trace.go#L758), which has proceeded to park_m. There, it runs m.waitunlockfn, which calls traceReader (https://github.com/golang/go/blob/go1.24.2/src/runtime/trace.go#L765), which calls traceReaderAvailable, which consults trace.shutdown (https://github.com/golang/go/blob/go1.24.2/src/runtime/trace.go#L938), which is true (https://github.com/golang/go/blob/go1.24.2/src/runtime/trace.go#L432). It looks to me like that would cause gopark to refuse to park the trace reader goroutine.
But because the world is being stopped, the goroutine that's running StopTrace has been put to sleep, so won't generate any new data for the trace reader goroutine and also won't proceed with the trace shutdown process?
Goroutine 131 - User: src/runtime/sys_linux_arm64.s:138 runtime.usleep (0x2abc644) (thread 17) [preempted]
0 0x0000000002abc644 in runtime.usleep
at src/runtime/sys_linux_arm64.s:138
1 0x0000000002a8dcb8 in runtime.sighandler
at GOROOT/src/runtime/signal_unix.go:826
2 0x0000000002a8d098 in runtime.sigtrampgo
at GOROOT/src/runtime/signal_unix.go:490
3 0x0000000002a7d6a4 in internal/runtime/atomic.(*Uint32).CompareAndSwap
at GOROOT/src/internal/runtime/atomic/types.go:236
4 0x0000000002a7d6a4 in runtime.casgstatus
at GOROOT/src/runtime/proc.go:1211
5 0x0000000002a7d6a4 in runtime.park_m
at GOROOT/src/runtime/proc.go:4112
6 0x0000000002ab9314 in runtime.mcall
at src/runtime/asm_arm64.s:193
7 0x0000000002a9afa8 in runtime.gopark
at GOROOT/src/runtime/proc.go:435
8 0x0000000002a9afa8 in runtime.ReadTrace
at GOROOT/src/runtime/trace.go:758
9 0x0000000002f91bec in runtime/trace.Start.func1
at GOROOT/src/runtime/trace/trace.go:130
10 0x0000000002abb854 in runtime.goexit
at src/runtime/asm_arm64.s:1223
Comment From: rhysh
From @nsrip-dd 's Runtime PGO debug output:
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../../../../sdk/go1.24.2/src/runtime/trace.go:758:9 in function runtime.ReadTrace
And after that (inlined) gopark call is goto top (https://github.com/golang/go/blob/go1.24.2/src/runtime/trace.go#L778). The only other function call (on the curg stack) is systemstack, which is assembly so (I think) wouldn't have a morestack opportunity. And runtime.ReadTrace is in the runtime package which would prevent async preemption (though I'm not sure if that matters for STWs.)
Without a morestack check on the gopark call, there's now no preemption point.
And park_m doesn't consult gp.preempt before calling execute, and execute clears that flag.
Comment From: mknyszek
Wow, nice catch @rhysh!
Comment From: mknyszek
So, I have a WIP change that adds an explicit preemption point, but I'm wondering if having traceReaderAvailable check trace.shutdown was a mistake. That was ported over from the old code IIRC, but the first condition, trace.flushedGen.Load() == trace.readerGen.Load() should be a sufficient signal to get the trace reader to kick its but into gear to finish up when the trace is shutting down (since a shutdown is just a fancy kind of traceAdvance).
Comment From: gopherbot
Change https://go.dev/cl/680738 mentions this issue: runtime: don't let readTrace spin on trace.shutdown
Comment From: gopherbot
Change https://go.dev/cl/680737 mentions this issue: runtime: add explicit preemption point in readTrace
Comment From: nsrip-dd
Thanks @rhysh and @mknyszek!
This program seems to reproduce the problem reliably (playground link):
package main
import (
"fmt"
"io"
"runtime"
"runtime/trace"
"time"
)
func main() {
go func() {
for {
// stop the world a lot
runtime.ReadMemStats(new(runtime.MemStats))
}
}()
go func() {
t := time.NewTicker(time.Second)
for range t.C {
fmt.Println("still alive")
}
}()
go func() {
for {
trace.Start(io.Discard)
trace.Stop()
}
}()
// block forever
select {}
}
I built it with PGO enabled using one of the affected profiles. I ran go tool pprof -proto -focus=runtime.gopark default.pgo > gopark.go to just get gopark frames. GitHub won't let me attach the profile to this comment. I'll see if I can store it somewhere for sharing. Then I built it and confirmed runtime.gopark gets inlined into runtime.ReadTrace:
% go1.24.2 build -gcflags=runtime=-d=pgodebug=1 -pgo=gopark.pgo |& grep ReadTrace
hot-budget check allows inlining for call runtime.gopark (cost 221) at ../../../sdk/go1.24.2/src/runtime/trace.go:758:9 in function runtime.ReadTrace
Then I ran the program. I never saw "still alive" print. If I removed tracing, I did see it. Here's a stack dump when the program appears to be stuck:
Stack dump
SIGQUIT: quit
PC=0x19f95f3cc m=0 sigcode=0
goroutine 0 gp=0x104ef27a0 m=0 mp=0x104ef2e00 [idle]:
runtime.pthread_cond_wait(0x104ef3358, 0x104ef3318)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/sys_darwin.go:547 +0x20 fp=0x16b086740 sp=0x16b086710 pc=0x104dcd730
runtime.semasleep(0xffffffffffffffff)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/os_darwin.go:72 +0x84 fp=0x16b0867a0 sp=0x16b086740 pc=0x104dab104
runtime.notesleep(0x104ef2f40)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/lock_sema.go:62 +0xb8 fp=0x16b0867e0 sp=0x16b0867a0 pc=0x104d871b8
runtime.mPark(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:1887
runtime.stopm()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:2907 +0x84 fp=0x16b086810 sp=0x16b0867e0 pc=0x104db5884
runtime.gcstopm()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:3218 +0xf8 fp=0x16b086840 sp=0x16b086810 pc=0x104db6448
runtime.findRunnable()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:3281 +0x2b0 fp=0x16b0869b0 sp=0x16b086840 pc=0x104db6930
runtime.schedule()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:4017 +0x94 fp=0x16b0869f0 sp=0x16b0869b0 pc=0x104db8b44
runtime.goschedImpl(0x14000003880, 0x0?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:4175 +0x1f4 fp=0x16b086a40 sp=0x16b0869f0 pc=0x104db9254
runtime.gopreempt_m(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:4192
runtime.newstack()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/stack.go:1074 +0x2cc fp=0x16b086b80 sp=0x16b086a40 pc=0x104dc8e4c
runtime.morestack()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:342 +0x70 fp=0x16b086b80 sp=0x16b086b80 pc=0x104de62e0
goroutine 1 gp=0x140000021c0 m=nil [select (no cases)]:
runtime.gopark(0x104dbb200?, 0x104e685a8?, 0xc0?, 0x21?, 0x104e17ebc?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435 +0xc8 fp=0x1400007cef0 sp=0x1400007ced0 pc=0x104de1bc8
runtime.block()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/select.go:104 +0x30 fp=0x1400007cf20 sp=0x1400007cef0 pc=0x104dc3530
main.main()
/Users/nick.ripley/sandbox/go/issue-74045/main.go:31 +0x40 fp=0x1400007cf40 sp=0x1400007cf20 pc=0x104e17ec0
runtime.main()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:283 +0x284 fp=0x1400007cfd0 sp=0x1400007cf40 pc=0x104db0664
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400007cfd0 sp=0x1400007cfd0 pc=0x104de8554
goroutine 2 gp=0x14000002700 m=nil [force gc (idle)]:
runtime.gopark(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435
runtime.goparkunlock(0x0?, 0x0?, 0x0?, 0x0?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:441 +0xd0 fp=0x14000050fa0 sp=0x14000050f80 pc=0x104db0b50
runtime.forcegchelper()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:348 +0xac fp=0x14000050fd0 sp=0x14000050fa0 pc=0x104db09ac
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x14000050fd0 sp=0x14000050fd0 pc=0x104de8554
created by runtime.init.7 in goroutine 1
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:336 +0x24
goroutine 3 gp=0x14000002c40 m=nil [GC sweep wait]:
runtime.gopark(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435
runtime.goparkunlock(0x0?, 0x0?, 0x0?, 0x0?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:441 +0xd0 fp=0x14000051770 sp=0x14000051750 pc=0x104db0b50
runtime.bgsweep(0x14000068000)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgcsweep.go:276 +0x94 fp=0x140000517b0 sp=0x14000051770 pc=0x104d9c354
runtime.gcenable.gowrap1()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgc.go:204 +0x28 fp=0x140000517d0 sp=0x140000517b0 pc=0x104d904e8
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x140000517d0 sp=0x140000517d0 pc=0x104de8554
created by runtime.gcenable in goroutine 1
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgc.go:204 +0x6c
goroutine 4 gp=0x14000002e00 m=nil [GC scavenge wait]:
runtime.gopark(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435
runtime.goparkunlock(0x14000068000?, 0x68?, 0xfa?, 0x1?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:441 +0xd0 fp=0x14000051f60 sp=0x14000051f40 pc=0x104db0b50
runtime.(*scavengerState).park(0x104ef2660)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgcscavenge.go:425 +0x50 fp=0x14000051f90 sp=0x14000051f60 pc=0x104d99e70
runtime.bgscavenge(0x14000068000)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgcscavenge.go:653 +0x44 fp=0x14000051fb0 sp=0x14000051f90 pc=0x104d9a3a4
runtime.gcenable.gowrap2()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgc.go:205 +0x28 fp=0x14000051fd0 sp=0x14000051fb0 pc=0x104d90488
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x14000051fd0 sp=0x14000051fd0 pc=0x104de8554
created by runtime.gcenable in goroutine 1
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mgc.go:205 +0xac
goroutine 5 gp=0x14000003340 m=nil [finalizer wait]:
runtime.gopark(0x140000505b8?, 0x104de2644?, 0x1?, 0x5?, 0x104debb94?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435 +0xc8 fp=0x14000050590 sp=0x14000050570 pc=0x104de1bc8
runtime.runfinq()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mfinal.go:196 +0x108 fp=0x140000507d0 sp=0x14000050590 pc=0x104d8f4e8
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x140000507d0 sp=0x140000507d0 pc=0x104de8554
created by runtime.createfing in goroutine 1
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mfinal.go:166 +0x80
goroutine 6 gp=0x14000003500 m=2 mp=0x14000056808 [stopping the world]:
runtime.systemstack_switch()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:201 +0x8 fp=0x14000184860 sp=0x14000184850 pc=0x104de6158
runtime.stopTheWorld(0x7)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:1448 +0xbc fp=0x140001848c0 sp=0x14000184860 pc=0x104db2f2c
runtime.ReadMemStats(0x14000184938)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/mstats.go:355 +0x28 fp=0x14000184920 sp=0x140001848c0 pc=0x104da8d48
main.main.func1()
/Users/nick.ripley/sandbox/go/issue-74045/main.go:15 +0x4c fp=0x14000185fd0 sp=0x14000184920 pc=0x104e17f1c
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x14000185fd0 sp=0x14000185fd0 pc=0x104de8554
created by main.main in goroutine 1
/Users/nick.ripley/sandbox/go/issue-74045/main.go:12 +0x24
goroutine 7 gp=0x140000036c0 m=nil [chan receive]:
runtime.gopark(0x1400008e080?, 0x104e055f0?, 0x80?, 0xe0?, 0x3156d30319547?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435 +0xc8 fp=0x14000078eb0 sp=0x14000078e90 pc=0x104de1bc8
runtime.chanrecv(0x1400008e000, 0x14000078f98, 0x1)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:664 +0x42c fp=0x14000078f30 sp=0x14000078eb0 pc=0x104d8289c
runtime.chanrecv2(0x3b9aca00?, 0x0?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:511 +0x14 fp=0x14000078f60 sp=0x14000078f30 pc=0x104d82454
main.main.func2()
/Users/nick.ripley/sandbox/go/issue-74045/main.go:20 +0x84 fp=0x14000078fd0 sp=0x14000078f60 pc=0x104e17fb4
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x14000078fd0 sp=0x14000078fd0 pc=0x104de8554
created by main.main in goroutine 1
/Users/nick.ripley/sandbox/go/issue-74045/main.go:18 +0x30
goroutine 8 gp=0x14000003880 m=nil [runnable]:
runtime.step({0x104ea80eb?, 0x1409d?, 0x1409d?}, 0x1400007b968?, 0x1400007b944?, 0x1?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/symtab.go:1248 +0x1c8 fp=0x1400007b900 sp=0x1400007b900 pc=0x104dccb88
runtime.pcvalue({0x3?, 0x104dcb438?}, 0x24fab, 0x104dd7bf4, 0x0)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/symtab.go:1045 +0x1ec fp=0x1400007b9d0 sp=0x1400007b900 pc=0x104dcc16c
runtime.funcline1({0x104e75afe?, 0x104e75afe?}, 0x104dd7bf4, 0x0)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/symtab.go:1160 +0x5c fp=0x1400007ba20 sp=0x1400007b9d0 pc=0x104de42bc
runtime.(*Frames).Next(0x14000112000)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/symtab.go:197 +0x630 fp=0x1400007bbb0 sp=0x1400007ba20 pc=0x104dcb7a0
runtime.makeTraceFrames(0x1, {0x14000110000, 0x4, 0x80})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/tracestack.go:210 +0xd0 fp=0x1400007bd30 sp=0x1400007bbb0 pc=0x104dd9cb0
runtime.dumpStacksRec(0x10531c038, {{0x0?, 0x1400007be08?}, 0x8?, 0x1400007bdf8?}, {0x14000110000, 0x80, 0x80})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/tracestack.go:165 +0xb4 fp=0x1400007bdc0 sp=0x1400007bd30 pc=0x104dd9764
runtime.(*traceStackTable).dump(0x104ef3710, 0x1)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/tracestack.go:153 +0x78 fp=0x1400007be10 sp=0x1400007bdc0 pc=0x104dd9648
runtime.traceAdvance(0x1)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:589 +0x590 fp=0x1400007bf70 sp=0x1400007be10 pc=0x104de5030
runtime.StopTrace(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:318
runtime/trace.Stop()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace/trace.go:148 +0xd8 fp=0x1400007bfb0 sp=0x1400007bf70 pc=0x104e17da8
main.main.func3()
/Users/nick.ripley/sandbox/go/issue-74045/main.go:27 +0x30 fp=0x1400007bfd0 sp=0x1400007bfb0 pc=0x104e18000
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400007bfd0 sp=0x1400007bfd0 pc=0x104de8554
created by main.main in goroutine 1
/Users/nick.ripley/sandbox/go/issue-74045/main.go:24 +0x3c
goroutine 17 gp=0x14000003a40 m=nil [chan receive]:
runtime.gopark(0x1400004e6a8?, 0x104dce6e0?, 0x3?, 0x0?, 0x14000003a40?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435 +0xc8 fp=0x1400004e680 sp=0x1400004e660 pc=0x104de1bc8
runtime.chanrecv(0x14000022150, 0x0, 0x1)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:664 +0x42c fp=0x1400004e700 sp=0x1400004e680 pc=0x104d8289c
runtime.chanrecv1(0x1400006c2a0?, 0x3156cfc60668f?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:506 +0x14 fp=0x1400004e730 sp=0x1400004e700 pc=0x104d82434
runtime.(*wakeableSleep).sleep(0x1400000c018, 0x5f5e100)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:1023 +0x98 fp=0x1400004e790 sp=0x1400004e730 pc=0x104dd13a8
runtime.traceStartReadCPU.func1()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/tracecpu.go:54 +0x48 fp=0x1400004e7d0 sp=0x1400004e790 pc=0x104dd6488
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400004e7d0 sp=0x1400004e7d0 pc=0x104de8554
created by runtime.traceStartReadCPU in goroutine 8
/Users/nick.ripley/sdk/go1.24.2/src/runtime/tracecpu.go:42 +0x154
goroutine 18 gp=0x14000003c00 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x3c?, 0x14000003c00?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435 +0xc8 fp=0x1400004ee90 sp=0x1400004ee70 pc=0x104de1bc8
runtime.chanrecv(0x140000222a0, 0x0, 0x1)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:664 +0x42c fp=0x1400004ef10 sp=0x1400004ee90 pc=0x104d8289c
runtime.chanrecv1(0x1400006c300?, 0x3156d320667ab?)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/chan.go:506 +0x14 fp=0x1400004ef40 sp=0x1400004ef10 pc=0x104d82434
runtime.(*wakeableSleep).sleep(0x1400000c030, 0x3b9aca00)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:1023 +0x98 fp=0x1400004efa0 sp=0x1400004ef40 pc=0x104dd13a8
runtime.(*traceAdvancerState).start.func1()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:960 +0x38 fp=0x1400004efd0 sp=0x1400004efa0 pc=0x104dd1258
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400004efd0 sp=0x1400004efd0 pc=0x104de8554
created by runtime.(*traceAdvancerState).start in goroutine 8
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:957 +0x174
goroutine 19 gp=0x14000003dc0 m=nil [running]:
runtime.gopark(...)
/Users/nick.ripley/sdk/go1.24.2/src/runtime/proc.go:435
runtime.ReadTrace()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace.go:758 +0x28 fp=0x1400004f790 sp=0x1400004f730 pc=0x104dd0ad8
runtime/trace.Start.func1()
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace/trace.go:130 +0x4c fp=0x1400004f7d0 sp=0x1400004f790 pc=0x104e17c0c
runtime.goexit({})
/Users/nick.ripley/sdk/go1.24.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400004f7d0 sp=0x1400004f7d0 pc=0x104de8554
created by runtime/trace.Start in goroutine 8
/Users/nick.ripley/sdk/go1.24.2/src/runtime/trace/trace.go:128 +0x11c
r0 0x104
r1 0x0
r2 0x8a400
r3 0x0
r4 0x0
r5 0xa0
r6 0x0
r7 0x0
r8 0x16b086628
r9 0x0
r10 0x200
r11 0x20000000202
r12 0x20000000202
r13 0x200
r14 0x0
r15 0x20000000200
r16 0x131
r17 0x20f8e0a78
r18 0x0
r19 0x104ef3318
r20 0x104ef3358
r21 0x20e870d60
r22 0x0
r23 0x0
r24 0x8a400
r25 0x8a401
r26 0x8a500
r27 0x104eed000
r28 0x104ef27a0
r29 0x16b0866a0
lr 0x19f99e0e0
sp 0x16b086610
pc 0x19f95f3cc
fault 0x19f95f3cc
I built the same program with https://go-review.googlesource.com/c/go/+/680738 applied and it worked as expected, even with runtime.gopark still inlined. Ditto for https://go-review.googlesource.com/c/go/+/680737/2. So it seems like either fix is good!
We can see about applying the patches to our build. As a workaround, we can also temporarily apply a similar technique to https://github.com/DataDog/datadog-pgo/pull/4 and scrub runtime.gopark from the profiles we use for PGO.
Comment From: nsrip-dd
Okay, turns out Github doesn't like .pgo but it's okay with .gz. Here's an anonymized profile that triggers the deadlock (livelock?) with the above program.
Comment From: mknyszek
I think we might want to consider backporting this, but I'd like to let the fix soak in the release candidate for a bit.
@nsrip-dd Thanks for the reproducer! I avoided writing a regression test based on preemption points because the mechanics of a test that specifically inlined gopark feels like overfitting -- I don't think it's likely to catch a regression in this specific issue in the future.
But it's also weird. If we consider spinning on shutdown the true bug here, then that's a different logic bug that's more fundamental to the trace design. This is trickier to test too -- I'm not sure how to write a non-flaky version of this test.
That being said, I think we may want to think of a way to more generally test the removal of synchronous preemption points, specifically in the runtime, as @rhysh suggests. Is it silly to suggest that we try removing all synchronous preemption points in the runtime? This would be the easiest thing to do. Naively, where the runtime is preemptible, is mostly for reducing preemption latency. In general, calls into the runtime, as well as background goroutines which all eventually block on some condition, all guarantee some kind of progress. I'm sure something I'm not thinking of would break, but I bet we'd find more real bugs. Such a mode would be better at catching whole classes of issues like this.
I feel bad landing the fix without a test, so I'm going to spend some time thinking about removing preemption points from the runtime. I think this might be as simple as having newstack ignore gp.preempt if the current function we're in is in the runtime.