Go version

go version go1.24.4 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE='auto'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/skatyal/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/skatyal/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/2x/v_msrx2s0wx6j4m7hyqt208c0000gn/T/go-build2398486118=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD=''
GOMODCACHE='/Users/skatyal/go/pkg/mod'
GONOPROXY='github.com/calculi-corp'
GONOSUMDB='github.com/calculi-corp'
GOOS='darwin'
GOPATH='/Users/skatyal/go'
GOPRIVATE='github.com/calculi-corp'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.24.1/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/skatyal/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.24.1/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

While trying to capture a CPU profile with the /debug/pprof/profile endpoint, the service panics and returns the error - ** execution is paused because your program is panicking ** To continue the execution please connect your client to the debugger. Stack trace: 0 0x00000000000522c0 in runtime.fatalpanic at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/panic.go:1301 1 0x0000000000087a0c in runtime.gopanic at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/panic.go:811 2 0x00000000000509a8 in runtime.panicmem at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/panic.go:262 3 0x0000000000089ff0 in runtime.sigpanic at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/signal_unix.go:925 4 0x00000000009a1588 in runtime/pprof.(*profileBuilder).build.func1 at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/proto.go:370 5 0x0000000000000075 in ??? at ?:-1 6 0x00000000009a1344 in runtime/pprof.(*profileBuilder).build at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/proto.go:378 7 0x00000040007bcd00 in ??? at ?:-1 8 0x000000000099e2a4 in runtime/pprof.StartCPUProfile.gowrap2 at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:856 9 0x00000040007bcc60 in ??? at ?:-1 10 0x000000000008f8f4 in runtime.goexit at /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/asm_arm64.s:1223 error: Undefined return address at 0x8f8f4 (truncated)

Similarly, hitting goroutines profile with debug mode /debug/pprof/goroutine?debug=1 give error -

runtime error: makeslice: cap out of range: goroutine 901 [running]:\nruntime/debug.Stack()\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/debug/stack.go:26 +0x64\ngithub.com/calculi-corp/grpc-server.(*Server).runHandler.func1()\n /Users/skatyal/go/pkg/mod/github.com/calculi-corp/grpc-server@v0.0.0-20250811065309-4390ca9e8354/server.go:518 +0x58\npanic({0x1468020?, 0x1888970?})\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/panic.go:792 +0xf0\nruntime/pprof.(*labelMap).String(0x4000800090)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/label.go:48 +0x54\nruntime/pprof.printCountProfile.func1({0x4000da02d0, 0x11, 0x11}, 0x4000800090)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:472 +0x18c\nruntime/pprof.printCountProfile({0xffff50ab56a0, 0x4000c582c0}, 0x1, {0x16c4969, 0x9}, {0x18a1450, 0x400130ad80})\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:481 +0x244\nruntime/pprof.writeRuntimeProfile({0xffff50ab56a0, 0x4000c582c0}, 0x1, {0x16c4969, 0x9}, 0x17a39f0)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:804 +0x230\nruntime/pprof.writeGoroutine({0xffff50ab56a0, 0x4000c582c0}, 0x1)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:755 +0x84\nruntime/pprof.(*Profile).WriteTo(0x2369380, {0xffff50ab56a0, 0x4000c582c0}, 0x1)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:377 +0x7c\nnet/http/pprof.handler.ServeHTTP({0x4000527d0d, 0x9}, {0x18a0960, 0x4000c582c0}, 0x4000365900)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/net/http/pprof/pprof.go:272 +0x3f0\nnet/http/pprof.Index({0x18a0960, 0x4000c582c0}, 0x4000365900)\n /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/net/http/pprof/pprof.go:389 +0xcc\ngithub.com/calculi-corp/grpc-server.(*pprofHandler).ServeHTTP(0x400007a3a8, {0x18a0960, 0x4000c582c0}, 0x4000365900)\n /Users/skatyal/go/pkg/mod/github.com/calculi-corp/grpc-server@v0.0.0-20250811065309-4390ca9e8354/server.go:894 +0x138\nnet/http.HandlerFunc.ServeHTTP(0x400078a8b0, {0x18a0960, 0x4000c582c0}, 0x4000365900)\n

It looks like an issue in pprof label handling, any way to resolve this panic?

What did you see happen?

Panic, attached logs in description

What did you expect to see?

The CPU profiling without panic

Comment From: seankhliao

I think we need a reproducer first, otherwise it looks more likely to be an issue in your program

Comment From: gabyhelp

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: saumeya-cloudbees

I think we need a reproducer first, otherwise it looks more likely to be an issue in your program

Thanks for the response, sure let me try getting a local reproducer for this.

Comment From: rhysh

Thanks for opening an issue, and thanks for including the behavior during the goroutine profile.

The goroutine profile failure seems due to corruption in the header of the slice that holds labels/tags in the runtime.g struct. That list is prepared via calls to runtime/pprof.Labels and runtime/pprof.WithLabels. Those calls generate the slice header and attach it to a context.Context value. The slice header isn't read/validated again until a/ formatting a goroutine profile for display (after collecting it), or b/ generating the pprof-format output of the CPU profile (when stopping it after collection). The runtime/pprof.SetGoroutineLabels call (also used by Do) doesn't inspect the labels.

The value of the strings themselves can't explain the goroutine profile crash.

I'd take a close look at any code that has a runtime/pprof.LabelSet value (might not be much of this, between the Labels and WithLabels calls), and at code that tries to interact with context.Context internals (via unsafe). It's possible that the corruption takes place after the labels are set on the runtime.g struct, but that seems less likely to me. (Why corruption there — an unsafe.Pointer that is a *[]struct{string,string} — and not on data that's pointed to by other G fields?)

Maybe that will help with a smaller / shareable reproducer, @saumeya-cloudbees .

The CPU profile encounters problems in a for/range loop: https://github.com/golang/go/blob/go1.24.4/src/runtime/pprof/proto.go#L370

But the Goroutine profile's failure to make a new slice is a bit clearer: https://github.com/golang/go/blob/go1.24.4/src/runtime/pprof/label.go#L48

That one means the runtime/pprof.labelMap.list has a non-sensical len. The length of the []struct{string,string} is negative, or would make the resulting []string not fit in memory (even though it would be half the size of the []struct{string,string}).

Here's the re-formatted stack trace:

runtime error: makeslice: cap out of range: goroutine 901 [running]: runtime/debug.Stack() /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/debug/stack.go:26 +0x64 github.com/calculi-corp/grpc-server.(*Server).runHandler.func1() /Users/skatyal/go/pkg/mod/github.com/calculi-corp/grpc-server@v0.0.0-20250811065309-4390ca9e8354/server.go:518 +0x58 panic({0x1468020?, 0x1888970?}) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/panic.go:792 +0xf0 runtime/pprof.(*labelMap).String(0x4000800090) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/label.go:48 +0x54 runtime/pprof.printCountProfile.func1({0x4000da02d0, 0x11, 0x11}, 0x4000800090) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:472 +0x18c runtime/pprof.printCountProfile({0xffff50ab56a0, 0x4000c582c0}, 0x1, {0x16c4969, 0x9}, {0x18a1450, 0x400130ad80}) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:481 +0x244 runtime/pprof.writeRuntimeProfile({0xffff50ab56a0, 0x4000c582c0}, 0x1, {0x16c4969, 0x9}, 0x17a39f0) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:804 +0x230 runtime/pprof.writeGoroutine({0xffff50ab56a0, 0x4000c582c0}, 0x1) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:755 +0x84 runtime/pprof.(*Profile).WriteTo(0x2369380, {0xffff50ab56a0, 0x4000c582c0}, 0x1) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/runtime/pprof/pprof.go:377 +0x7c net/http/pprof.handler.ServeHTTP({0x4000527d0d, 0x9}, {0x18a0960, 0x4000c582c0}, 0x4000365900) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/net/http/pprof/pprof.go:272 +0x3f0 net/http/pprof.Index({0x18a0960, 0x4000c582c0}, 0x4000365900) /Users/skatyal/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.darwin-arm64/src/net/http/pprof/pprof.go:389 +0xcc github.com/calculi-corp/grpc-server.(*pprofHandler).ServeHTTP(0x400007a3a8, {0x18a0960, 0x4000c582c0}, 0x4000365900) /Users/skatyal/go/pkg/mod/github.com/calculi-corp/grpc-server@v0.0.0-20250811065309-4390ca9e8354/server.go:894 +0x138 net/http.HandlerFunc.ServeHTTP(0x400078a8b0, {0x18a0960, 0x4000c582c0}, 0x4000365900)