$ go version
go version go1.23rc2 linux/386

How to reproduce: * build anything that calls fmt.Println (for example buildtest) on linux/386 with go 1.23rc2 passing -gcflags='all=-N -l' * run the executable * example crash:

runtime: newstack at runtime.lockWithRankMayAcquire+0x18 sp=0x9c64bf4 stack=[0x9c64000, 0x9c65000]
        morebuf={pc:0x8087426 sp:0x9c64bf8 lr:0x0}
        sched={pc:0x8054688 sp:0x9c64bf4 lr:0x0 ctxt:0x0}
runtime.casgstatus(0x9c06128, 0x2, 0x3)
        /home/d/go-tip/src/runtime/proc.go:1185 +0x56 fp=0x9c64c30 sp=0x9c64bf8 pc=0x8087426
runtime.reentersyscall(0x80e5c70, 0x9c64c74, 0x0)
        /home/d/go-tip/src/runtime/proc.go:4410 +0x89 fp=0x9c64c64 sp=0x9c64c30 pc=0x808da19
runtime.entersyscall()
        /home/d/go-tip/src/runtime/proc.go:4479 +0x23 fp=0x9c64c74 sp=0x9c64c64 pc=0x80b9693
syscall.Syscall(0x4, 0x1, 0x9c120d0, 0xd)
        /home/d/go-tip/src/syscall/syscall_linux.go:73 +0x20 fp=0x9c64cb8 sp=0x9c64c74 pc=0x80e5c70
syscall.write(0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/syscall/zsyscall_linux_386.go:964 +0x96 fp=0x9c64d04 sp=0x9c64cb8 pc=0x80e57d6
syscall.Write(0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/syscall/syscall_unix.go:211 +0x65 fp=0x9c64d3c sp=0x9c64d04 pc=0x80e5395
internal/poll.ignoringEINTRIO(0x8117ae8, 0x1, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/internal/poll/fd_unix.go:745 +0x6c fp=0x9c64d88 sp=0x9c64d3c pc=0x80e8a6c
internal/poll.(*FD).Write(0x9c460c0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/internal/poll/fd_unix.go:381 +0x299 fp=0x9c64e80 sp=0x9c64d88 pc=0x80e8689
os.(*File).write(0x9c142f0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/os/file_posix.go:46 +0x6f fp=0x9c64ec0 sp=0x9c64e80 pc=0x80e9e5f
os.(*File).Write(0x9c142f0, {0x9c120d0, 0xd, 0x10})
        /home/d/go-tip/src/os/file.go:195 +0xad fp=0x9c64f08 sp=0x9c64ec0 pc=0x80e9a5d
fmt.Fprintln({0x811d6d4, 0x9c142f0}, {0x9c64fb4, 0x1, 0x1})
        /home/d/go-tip/src/fmt/print.go:305 +0xa5 fp=0x9c64f50 sp=0x9c64f08 pc=0x80ee995
fmt.Println({0x9c64fb4, 0x1, 0x1})
        /home/d/go-tip/src/fmt/print.go:314 +0x71 fp=0x9c64f8c sp=0x9c64f50 pc=0x80eea61
main.main()
        /home/d/delve/_fixtures/buildtest/main.go:6 +0x8a fp=0x9c64fc0 sp=0x9c64f8c pc=0x80f399a
runtime.main()
        /home/d/go-tip/src/runtime/proc.go:272 +0x265 fp=0x9c64ff0 sp=0x9c64fc0 pc=0x80851a5
runtime.goexit({})
        /home/d/go-tip/src/runtime/asm_386.s:1393 +0x1 fp=0x9c64ff4 sp=0x9c64ff0 pc=0x80be5b1
fatal error: runtime: stack split at bad time

Comment From: gabyhelp

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: gopherbot

Change https://go.dev/cl/599675 mentions this issue: runtime: mark lockWithRankMayAcquire as nosplit

Comment From: cuonglm

git bisect points to https://go-review.googlesource.com/c/go/+/571056

The CL call lockWithRankMayAcquire during casgstatus, which is marked as nosplit. Normally, it's not problem because this function is empty when lockrank is off. However, when inlining is disable, the function is not inlined anymore, and requiring newstack when calling it -> crashing the runtime.

Comment From: rhysh

Thanks for the report and the fix. I agree with the fix: a nosplit annotation is required there.

It's a big strange to me that GOARCH=386 behaves differently here than GOARCH=amd64. Building with -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' to see if there are other problems to shake out, I see that GOARCH=amd64 and GOARCH=arm64 get trivial implementations of runtime.lockWithRankMayAcquire (nothing more than RET) but that GOARCH=386 has the usual stack growth function preamble.

Lack of nosplit annotation was a bug, the fix is right, what remains is to see if there's anything to be learned from why the bug was hidden except on GOARCH=386.

$ ./bin/go version
go version go1.23rc2 linux/amd64

$ cat /tmp/hello.go 
// run -gcflags='all=-N -l'

// Copyright 2024 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

import "os"

func main() {
    os.OpenFile(os.DevNull, os.O_WRONLY, 0)
}

$ GOOS=linux GOARCH=arm64 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:46    0x1c680         d65f03c0        RET         
  lockrank_off.go:46    0x1c684         00000000        ?           
  lockrank_off.go:46    0x1c688         00000000        ?           
  lockrank_off.go:46    0x1c68c         00000000        ?

$ GOOS=linux GOARCH=amd64 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:46    0x40cca0        c3          RET     

$ GOOS=linux GOARCH=386 ./bin/go build -o /tmp/hello -asmflags='all=-d=maymorestack=runtime.mayMoreStackPreempt' -gcflags='all=-N -l -d=maymorestack=runtime.mayMoreStackPreempt' /tmp/hello.go 
$ ./bin/go tool objdump -s lockWithRankMayAcquire /tmp/hello
TEXT runtime.lockWithRankMayAcquire(SB) /home/rhysh/sdk/go1.23/src/runtime/lockrank_off.go
  lockrank_off.go:45    0x8053d90       e87b1e0600      CALL runtime.mayMoreStackPreempt(SB)    
  lockrank_off.go:45    0x8053d95       658b0d00000000      MOVL GS:0, CX               
  lockrank_off.go:45    0x8053d9c       8b89fcffffff        MOVL 0xfffffffc(CX), CX         
  lockrank_off.go:45    0x8053da2       3b6108          CMPL SP, 0x8(CX)            
  lockrank_off.go:45    0x8053da5       7601            JBE 0x8053da8               
  lockrank_off.go:46    0x8053da7       c3          RET                 
  lockrank_off.go:45    0x8053da8       e8039f0600      CALL runtime.morestack_noctxt(SB)   
  lockrank_off.go:45    0x8053dad       ebe6            JMP 0x8053d95

Comment From: cuonglm

Lack of nosplit annotation was a bug, the fix is right, what remains is to see if there's anything to be learned from why the bug was hidden except on GOARCH=386.

Not only 386, this also happens for other platforms where reg ABI is not supported. On these platforms, we need to setup stack for function arguments:

func lockWithRankMayAcquire(l *mutex, rank lockRank) {}

Comment From: rhysh

So that's part of how regabi (or lack of it) looks, got it. Thanks @cuonglm .