Go version

N/A

Output of go env in your module/workspace:

N/A

What did you do?

I maintain the Go package for Alpine Linux, this package builds Go from source using ./make.bash.

While upgrading from Go 1.21.6 to Go 1.22.0 we noticed that Go itself wouldn't build on our armv7 CI.

What did you see happen?

Go 1.22.0 builds successful on Alpine x86_64, x86, armhf (GOARM=6), ppc64le, aarch64 and riscv64. It only fails to build on armv7 (GOARM=7). Furthermore, Go 1.21.6 builds fine on our armv7 CI. Therefore, I believe this to be a regression in Go 1.22.0. The build failure message is not consistent, there is probably a race condition somewhere. However, it always fails either after Building Go toolchain2 using go_bootstrap and Go toolchain1 or Building Go toolchain3 using go_bootstrap and Go toolchain2.

Build logs of some of the failures we have seen so far (as I said, the exact error message differs):

Any idea what might be causing this? Any hints on debugging this further?

What did you expect to see?

A successful build of Go 1.22.0 on all architectures.

Comment From: dmitshur

CC @golang/arm.

Comment From: seankhliao

Are these bare metal or cross arch virtualized machines?

Comment From: cherrymui

Thank for the report. Would it be okay for you to do a bisection? Thanks.

Comment From: nmeum

Are these bare metal or cross arch virtualized machines?

These are armv7 Docker containers running on an aarch64 host.

Thank for the report. Would it be okay for you to do a bisection? Thanks.

I am currently somewhat short-pressed on time, but I will try to look into it.

Comment From: nmeum

The commit which introduced the regression is 66b8107a26e515bbe19855d358bdf12bd6326347 (CC: @randall77).

Reverting this commit fixes the build.

Comment From: cherrymui

It might be an interesting interaction between CL 525637 and CL 514907. Specifically, this code https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/work/gc.go;l=369-381 is defining assembler macros GOARM_x based on cfg.GOARM and expecting a value like "5", "6", or "7", but with CL 514907, it can also be "7,hardfloat" (which is probably the case if you set GOARM at make.bash), which caused the wrong macro to be defined, which in turn makes it possible to drop some memory barrier instructions. There is also run-time CPU feature detection based on the HWCAP, but for a Docker container running on ARM64, that could go off.

Could you try if CL https://go.dev/cl/562995 helps? Thanks.

Comment From: gopherbot

Change https://go.dev/cl/562995 mentions this issue: cmd/go: handle GOARM value with soft/hardfloat in defining assembly macros

Comment From: nmeum

Could you try if CL https://go.dev/cl/562995 helps? Thanks.

Still deadlocks with this patch.

Comment From: cherrymui

Thanks. I updated the CL. Could you try the new version of the CL? It seems we also need to define GOARM_x in cmd/dist.

Comment From: nmeum

Thanks for the patch. Unfortunately, it still deadlocks for me with the updated version of the CL.

Comment From: gopherbot

Change https://go.dev/cl/564855 mentions this issue: runtime/internal/atomic: correct GOARM=7 guard at a DMB instruction

Comment From: cherrymui

Thanks. Could you try https://go.dev/cl/564855 ? That should fix the build. (CL 562995 is probably good to have but is not required for a successful build.)

Comment From: cherrymui

@gopherbot please backport this to Go 1.22 release. This is a regression, causing runtime crashes with GOARM=7.

Comment From: gopherbot

Backport issue(s) opened: #65760 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

Comment From: gopherbot

Change https://go.dev/cl/567555 mentions this issue: [release-branch.go1.22] runtime/internal/atomic: correct GOARM=7 guard at a DMB instruction

Comment From: clausecker

@cherrymui It seems that your backport was incomplete. Commit e4ebd13f7a6f216f7ddbaa1f2a3c97fd10bc3d23 of CL562995 has not been backported, but seems to be required to fix the build. As is, Go 1.22.3 doesn't build on armv7 FreeBSD. Could you backport the missing commit, too?

Comment From: cherrymui

@clausecker thanks for reporting. I thought that CL 562995 is nice to have to but not a requirement: if the macro is not defined correctly it just builds a binary that is not optimized for GOARM=7. It is possible I missed something. What is the error you saw? Thanks.

Comment From: clausecker

@cherrymui I get crashes and hangs that look very similar to what @nmeum saw. A bisection led me to 6ecadb4d87dafebd42fa7e810faa1bf1a0e6f35d for the first commit where this happens (may be incorrect) and to e4ebd13f7a6f216f7ddbaa1f2a3c97fd10bc3d23 for where this behaviour ceases.

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278953 for where I track the build failure in our port.