Go version
go1.22.6 linux/arm64
Output of go env in your module/workspace:
n/a
What did you do?
I was running an application test suite against an arm64 K3s built with go1.22.6:
$ docker run --rm --entrypoint=k3s docker.io/rancher/k3s:v1.31.1-k3s1 --version
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6
What did you see happen?
kubelet reported that an exec probe failed with SIGSEGV:
Liveness probe failed: SIGSEGV: segmentation violation
…
goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]:
runtime.systemstack_switch()
runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228
runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c})
runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10
syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c
syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870
os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?})
os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4
…
full message
Liveness probe failed: SIGSEGV: segmentation violation
PC=0xffff804967e4 m=0 sigcode=1 addr=0x0
goroutine 0 gp=0xffff80950d00 m=0 mp=0xffff80951580 [idle]:
runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4
stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270)
0x0000ffffe77cbdf0: 0x0000ffff800dce40 <runtime.persistentalloc.func1+0x0000000000000030> 0x0000000000000000
0x0000ffffe77cbe00: 0xffffffffffffefe8 0x0000000000000080
0x0000ffffe77cbe10: 0xffffffffffffff80 0x0000ffff80951580
0x0000ffffe77cbe20: 0x0000004000035220 0x0000ffff3936eb80
0x0000ffffe77cbe30: 0xfffffffffffff800 0x0000ffffe77cbe68
0x0000ffffe77cbe40: 0x0000ffffe77cbe90 0x0000ffff80496388
0x0000ffffe77cbe50: 0x0000ffffe77cbe90 0x0000ffff80496710
0x0000ffffe77cbe60: 0x0000000000000002 0x0000aaaac2860130
0x0000ffffe77cbe70: 0x0000ffffe77cbe90 0x0000ffff80496710
0x0000ffffe77cbe80: 0x0000000000000002 0x0000ffff3936eb80
0x0000ffffe77cbe90: 0x0000ffffe77cbef0 0x0000ffff80496910
0x0000ffffe77cbea0: 0x0000000000000002 0x0000aaaac2860108
0x0000ffffe77cbeb0: 0x0000000000000023 0x0000ffff809b7000
0x0000ffffe77cbec0: 0x0000000000000023 0x0000000000003ff0
0x0000ffffe77cbed0: 0x0000000000000027 0x0000ffff8056d000
0x0000ffffe77cbee0: 0x0000000000000002 0x0000ffffe77cbf68
0x0000ffffe77cbef0: <0x0000ffffe77cbf50 0x0000ffff80496910
0x0000ffffe77cbf00: 0x0000000000000002 0x0000aaaac28600e0
0x0000ffffe77cbf10: 0x000000000000001f 0x0000ffff809b7000
0x0000ffffe77cbf20: 0x000000000000001f 0x0000000000001ff0
0x0000ffffe77cbf30: 0x0000000000000023 0x0000ffff8056d000
0x0000ffffe77cbf40: 0x0000000000000002 0x0000ffffe77cbfb8
0x0000ffffe77cbf50: 0x0000ffffe77cbfb0 0x0000ffff80496910
0x0000ffffe77cbf60: 0x0000000000000002 0x0000aaaac28600b8
0x0000ffffe77cbf70: 0x000000000000001b 0x0000ffff809b7000
0x0000ffffe77cbf80: 0x000000000000001b 0x0000000000000ff0
0x0000ffffe77cbf90: 0x000000000000001f 0x0000ffff8056d000
0x0000ffffe77cbfa0: 0x0000000000000002 0x0000ffffe77cbfd8
0x0000ffffe77cbfb0: 0x0000ffffe77cc010 0x0000ffff80496910
0x0000ffffe77cbfc0: 0x0000000000000002 0x0000aaaac2860090
0x0000ffffe77cbfd0: 0x0000000000000017 0x0000ffff809b7000
0x0000ffffe77cbfe0: 0x0000000000000017 0x00000000000007f0
runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4
stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270)
0x0000ffffe77cbdf0: 0x0000ffff800dce40 <runtime.persistentalloc.func1+0x0000000000000030> 0x0000000000000000
0x0000ffffe77cbe00: 0xffffffffffffefe8 0x0000000000000080
0x0000ffffe77cbe10: 0xffffffffffffff80 0x0000ffff80951580
0x0000ffffe77cbe20: 0x0000004000035220 0x0000ffff3936eb80
0x0000ffffe77cbe30: 0xfffffffffffff800 0x0000ffffe77cbe68
0x0000ffffe77cbe40: 0x0000ffffe77cbe90 0x0000ffff80496388
0x0000ffffe77cbe50: 0x0000ffffe77cbe90 0x0000ffff80496710
0x0000ffffe77cbe60: 0x0000000000000002 0x0000aaaac2860130
0x0000ffffe77cbe70: 0x0000ffffe77cbe90 0x0000ffff80496710
0x0000ffffe77cbe80: 0x0000000000000002 0x0000ffff3936eb80
0x0000ffffe77cbe90: 0x0000ffffe77cbef0 0x0000ffff80496910
0x0000ffffe77cbea0: 0x0000000000000002 0x0000aaaac2860108
0x0000ffffe77cbeb0: 0x0000000000000023 0x0000ffff809b7000
0x0000ffffe77cbec0: 0x0000000000000023 0x0000000000003ff0
0x0000ffffe77cbed0: 0x0000000000000027 0x0000ffff8056d000
0x0000ffffe77cbee0: 0x0000000000000002 0x0000ffffe77cbf68
0x0000ffffe77cbef0: <0x0000ffffe77cbf50 0x0000ffff80496910
0x0000ffffe77cbf00: 0x0000000000000002 0x0000aaaac28600e0
0x0000ffffe77cbf10: 0x000000000000001f 0x0000ffff809b7000
0x0000ffffe77cbf20: 0x000000000000001f 0x0000000000001ff0
0x0000ffffe77cbf30: 0x0000000000000023 0x0000ffff8056d000
0x0000ffffe77cbf40: 0x0000000000000002 0x0000ffffe77cbfb8
0x0000ffffe77cbf50: 0x0000ffffe77cbfb0 0x0000ffff80496910
0x0000ffffe77cbf60: 0x0000000000000002 0x0000aaaac28600b8
0x0000ffffe77cbf70: 0x000000000000001b 0x0000ffff809b7000
0x0000ffffe77cbf80: 0x000000000000001b 0x0000000000000ff0
0x0000ffffe77cbf90: 0x000000000000001f 0x0000ffff8056d000
0x0000ffffe77cbfa0: 0x0000000000000002 0x0000ffffe77cbfd8
0x0000ffffe77cbfb0: 0x0000ffffe77cc010 0x0000ffff80496910
0x0000ffffe77cbfc0: 0x0000000000000002 0x0000aaaac2860090
0x0000ffffe77cbfd0: 0x0000000000000017 0x0000ffff809b7000
0x0000ffffe77cbfe0: 0x0000000000000017 0x00000000000007f0
goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]:
runtime.systemstack_switch()
runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228
runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c})
runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10
syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c
syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870
os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?})
os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4
github.com/opencontainers/runc/libcontainer.populateProcessEnvironment({0x400017a200?, 0xffff805b5440?, 0x400004a0f8?})
github.com/opencontainers/runc/libcontainer/init_linux.go:138 +0xe0 fp=0x400018fb80 sp=0x400018fb00 pc=0xffff80401190
github.com/opencontainers/runc/libcontainer.newContainerInit({0x4000028017, 0x5}, 0x400004a0f0, 0x0, 0xffffffffffffffff, 0x4, {0x0, 0x0, 0x0})
github.com/opencontainers/runc/libcontainer/init_linux.go:85 +0xb4 fp=0x400018fbe0 sp=0x400018fb80 pc=0xffff80400da4
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x0?)
github.com/opencontainers/runc/libcontainer/factory_linux.go:334 +0x3d0 fp=0x400018fdc0 sp=0x400018fbe0 pc=0xffff80400310
main.init.0()
github.com/opencontainers/runc/init.go:36 +0x194 fp=0x400018fe10 sp=0x400018fdc0 pc=0xffff8047c7b4
runtime.doInit1(0xffff809399e0)
runtime/proc.go:7176 +0xd4 fp=0x400018ff40 sp=0x400018fe10 pc=0xffff80119a44
runtime.doInit(...)
runtime/proc.go:7143
runtime.main()
runtime/proc.go:253 +0x344 fp=0x400018ffd0 sp=0x400018ff40 pc=0xffff80109bf4
runtime.goexit({})
runtime/asm_arm64.s:1222 +0x4 fp=0x400018ffd0 sp=0x400018ffd0 pc=0xffff8013e754
goroutine 2 gp=0x4000002c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:402 +0xc8 fp=0x4000046f90 sp=0x4000046f70 pc=0xffff80109f68
runtime.goparkunlock(...)
runtime/proc.go:408
runtime.forcegchelper()
runtime/proc.go:326 +0xb8 fp=0x4000046fd0 sp=0x4000046f90 pc=0xffff80109df8
runtime.goexit({})
runtime/asm_arm64.s:1222 +0x4 fp=0x4000046fd0 sp=0x4000046fd0 pc=0xffff8013e754
created by runtime.init.6 in goroutine 1
runtime/proc.go:314 +0x24
goroutine 3 gp=0x4000003180 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:402 +0xc8 fp=0x4000047760 sp=0x4000047740 pc=0xffff80109f68
runtime.goparkunlock(...)
runtime/proc.go:408
runtime.bgsweep(0x400006e000)
runtime/mgcsweep.go:278 +0xa0 fp=0x40000477b0 sp=0x4000047760 pc=0xffff800f54a0
runtime.gcenable.gowrap1()
runtime/mgc.go:203 +0x28 fp=0x40000477d0 sp=0x40000477b0 pc=0xffff800e96b8
runtime.goexit({})
runtime/asm_arm64.s:1222 +0x4 fp=0x40000477d0 sp=0x40000477d0 pc=0xffff8013e754
created by runtime.gcenable in goroutine 1
runtime/mgc.go:203 +0x6c
goroutine 4 gp=0x4000003340 m=nil [GC scavenge wait]:
runtime.gopark(0x400006e000?, 0xffff80555fe8?, 0x1?, 0x0?, 0x4000003340?)
runtime/proc.go:402 +0xc8 fp=0x4000047f60 sp=0x4000047f40 pc=0xffff80109f68
runtime.goparkunlock(...)
runtime/proc.go:408
runtime.(*scavengerState).park(0xffff80950a20)
runtime/mgcscavenge.go:425 +0x5c fp=0x4000047f90 sp=0x4000047f60 pc=0xffff800f2e9c
runtime.bgscavenge(0x400006e000)
runtime/mgcscavenge.go:653 +0x44 fp=0x4000047fb0 sp=0x4000047f90 pc=0xffff800f33e4
runtime.gcenable.gowrap2()
runtime/mgc.go:204 +0x28 fp=0x4000047fd0 sp=0x4000047fb0 pc=0xffff800e9658
runtime.goexit({})
runtime/asm_arm64.s:1222 +0x4 fp=0x4000047fd0 sp=0x4000047fd0 pc=0xffff8013e754
created by runtime.gcenable in goroutine 1
runtime/mgc.go:204 +0xac
goroutine 5 gp=0x4000003c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0xb8?, 0x65?, 0xffff80139b64?)
runtime/proc.go:402 +0xc8 fp=0x4000046580 sp=0x4000046560 pc=0xffff80109f68
runtime.runfinq()
runtime/mfinal.go:194 +0x108 fp=0x40000467d0 sp=0x4000046580 pc=0xffff800e8788
runtime.goexit({})
runtime/asm_arm64.s:1222 +0x4 fp=0x40000467d0 sp=0x40000467d0 pc=0xffff8013e754
created by runtime.createfing in goroutine 1
runtime/mfinal.go:164 +0x80
r0 0x0
r1 0x50
r2 0x0
r3 0x0
r4 0x2e
r5 0xffff394e0010
r6 0xaaaac2860130
r7 0x7fec
r8 0x0
r9 0xffff394e7ffc
r10 0x2f7273752f3a6e69
r11 0x73752f3a6e696273
r12 0x732f3a6e69622f72
r13 0xffffe77cc220
r14 0x2f
r15 0x30
r16 0x80
r17 0x400018f920
r18 0x0
r19 0x2
r20 0xaaaac2860108
r21 0x23
r22 0xffff809b7000
r23 0x23
r24 0x3ff0
r25 0x27
r26 0xffff8056dd20
r27 0x2
r28 0xffff80950d00
r29 0xffffe77cbef0
lr 0xffff80496910
sp 0xffffe77cbef0
pc 0xffff804967e4
fault 0x0
What did you expect to see?
No crash.
Perhaps the issue discussed in #59692 can also happen on arm64?
Comment From: gabyhelp
Related Issues and Documentation
- runtime: misc/cgo/test.TestSetgid panic with `unexpected return pc for runtime.sigpanic` on linux-arm64-packet since 2022-06-04 #53374 (closed)
- runtime: segmentation violation runtime.(*unwinder).next #66151 (closed)
- runtime: linux/arm64 crash in runtime.sigtrampgo #32912 (closed)
- fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
- cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)
- runtime: fatal error: unexpected signal during runtime execution #60783 (closed)
- runtime: frequent TestSegv failures on linux-mips64le-mengzhuo #50605 (closed)
- runtime: `TestSegv/Segv` failure with `unexpected return pc` on `openbsd-386-68` #52043
- runtime #60196 (closed)
- runtime/cgo: SIGSEGV: segmentation violation #6051 (closed)
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
Comment From: cherrymui
What is your host machine? Are you running docker as a VM?
It looks like this is crashing in the C setenv function. What are your libc and C toolchain? Thanks.
Comment From: mateusz834
See #63567
Comment From: mpx
As mentioned elsewhere (#63567), libc setenv(3) is not threadsafe - it's a fundamental flaw. Quite a few different libc calls can access the environment in parallel to setenv(3) and cause a segfault (eg, getaddrinfo(3)).
Some options to avoid the libc segfault:
- Avoid libc by disabling CGO (CGO_ENABLE=0 during compilation)
- Ensure your program never modifies the environment once there are multiple goroutines (eg, limit changes to func init).
Comment From: gopherbot
Timed out in state WaitingForInfo. Closing.
(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)