Our current builder (generously contributed by @zx2c4) is ready to be turned down. We need additional hardware to keep supporting this platform, and can re-use the infrastructure from #42604.
Comment From: gopherbot
Change https://golang.org/cl/344990 mentions this issue: dashboard: add windows-arm-10 builder
Comment From: gopherbot
Change https://golang.org/cl/345130 mentions this issue: dashboard: fix windows-arm-10 build failure
Comment From: toothrot
We're running into a 0xc000007b failure when attempting to run arm32 binaries in our arm64 windows installation. It could be either the version of windows we have installed, or a strange interaction with QEMU and HVM.
-e GOARM=7 -e GOARCH=arm -e GOHOSTARCH=arm:
go run ./cmd/gomote run -debug -e GOARM=7 -e GOARCH=arm -e GOHOSTARCH=arm user-rakoczy-windows-arm-10-0 src/make.bat
:: Running C:\workdir\src\make.bat with args ["C:\\workdir\\src\\make.bat"] and env ["GOBUILDEXIT=1" "=C:=C:\\golang" "ALLUSERSPROFILE=C:\\ProgramData" "APPDATA=C:\\Users\\gopher\\AppData\\Roaming" "CommonProgramFiles=C:\\Program Files\\Common Files" "CommonProgramFiles(Arm)=C:\\Program Files (Arm)\\Common Files" "CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files" "CommonProgramW6432=C:\\Program Files\\Common Files" "COMPUTERNAME=DESKTOP-0285N7P" "ComSpec=C:\\WINDOWS\\system32\\cmd.exe" "DriverData=C:\\Windows\\System32\\Drivers\\DriverData" "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer" "FPS_BROWSER_USER_PROFILE_STRING=Default" "GOROOT_BOOTSTRAP=C:\\workdir\\go1.4" "GO_BUILDER_ENV=host-windows-arm64-mini" "HOMEDRIVE=C:" "HOMEPATH=\\Users\\gopher" "LOCALAPPDATA=C:\\Users\\gopher\\AppData\\Local" "LOGONSERVER=\\\\DESKTOP-0285N7P" "NUMBER_OF_PROCESSORS=8" "OneDrive=C:\\Users\\gopher\\OneDrive" "OS=Windows_NT" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\gcc64\\bin" "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC" "PROCESSOR_ARCHITECTURE=ARM64" "PROCESSOR_IDENTIFIER=ARMv8 (64-bit) Family 8 Model 51 Revision 0, QEMU" "PROCESSOR_LEVEL=81" "PROCESSOR_REVISION=0000" "ProgramData=C:\\ProgramData" "ProgramFiles=C:\\Program Files" "ProgramFiles(Arm)=C:\\Program Files (Arm)" "ProgramFiles(x86)=C:\\Program Files (x86)" "ProgramW6432=C:\\Program Files" "PROMPT=$P$G" "PSModulePath=C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules" "PUBLIC=C:\\Users\\Public" "SESSIONNAME=Console" "SystemDrive=C:" "SystemRoot=C:\\WINDOWS" "TEMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "TMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "USERDOMAIN=DESKTOP-0285N7P" "USERDOMAIN_ROAMINGPROFILE=DESKTOP-0285N7P" "USERNAME=gopher" "USERPROFILE=C:\\Users\\gopher" "windir=C:\\WINDOWS" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=700ms" "WORKDIR=C:\\workdir" "GO_BUILDER_NAME=windows-arm-10" "GOARCH=arm" "GOARM=7" "GOHOSTARCH=arm" "GO_DISABLE_OUTBOUND_NETWORK=false"] in dir C:\workdir\src
Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
go tool dist: FAILED: C:\workdir\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 0xc000007b
Error running run: exit status 1
exit status 1
-e GOARM=7 -e GOARCH=arm:
go run ./cmd/gomote run -debug -e GOARM=7 -e GOARCH=arm -e "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\llvm-aarch64\\bin" user-rakoczy-windows-arm-10-0 src/all.bat
:: Running C:\workdir\src\all.bat with args ["C:\\workdir\\src\\all.bat"] and env ["GOBUILDEXIT=1" "=C:=C:\\golang" "ALLUSERSPROFILE=C:\\ProgramData" "APPDATA=C:\\Users\\gopher\\AppData\\Roaming" "CommonProgramFiles=C:\\Program Files\\Common Files" "CommonProgramFiles(Arm)=C:\\Program Files (Arm)\\Common Files" "CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files" "CommonProgramW6432=C:\\Program Files\\Common Files" "COMPUTERNAME=DESKTOP-0285N7P" "ComSpec=C:\\WINDOWS\\system32\\cmd.exe" "DriverData=C:\\Windows\\System32\\Drivers\\DriverData" "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer" "FPS_BROWSER_USER_PROFILE_STRING=Default" "GOROOT_BOOTSTRAP=C:\\workdir\\go1.4" "GO_BUILDER_ENV=host-windows-arm64-mini" "HOMEDRIVE=C:" "HOMEPATH=\\Users\\gopher" "LOCALAPPDATA=C:\\Users\\gopher\\AppData\\Local" "LOGONSERVER=\\\\DESKTOP-0285N7P" "NUMBER_OF_PROCESSORS=8" "OneDrive=C:\\Users\\gopher\\OneDrive" "OS=Windows_NT" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\gcc64\\bin" "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC" "PROCESSOR_ARCHITECTURE=ARM64" "PROCESSOR_IDENTIFIER=ARMv8 (64-bit) Family 8 Model 51 Revision 0, QEMU" "PROCESSOR_LEVEL=81" "PROCESSOR_REVISION=0000" "ProgramData=C:\\ProgramData" "ProgramFiles=C:\\Program Files" "ProgramFiles(Arm)=C:\\Program Files (Arm)" "ProgramFiles(x86)=C:\\Program Files (x86)" "ProgramW6432=C:\\Program Files" "PROMPT=$P$G" "PSModulePath=C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules" "PUBLIC=C:\\Users\\Public" "SESSIONNAME=Console" "SystemDrive=C:" "SystemRoot=C:\\WINDOWS" "TEMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "TMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "USERDOMAIN=DESKTOP-0285N7P" "USERDOMAIN_ROAMINGPROFILE=DESKTOP-0285N7P" "USERNAME=gopher" "USERPROFILE=C:\\Users\\gopher" "windir=C:\\WINDOWS" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=700ms" "WORKDIR=C:\\workdir" "GO_BUILDER_NAME=windows-arm-10" "GOARCH=arm" "GOARM=7" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\llvm-aarch64\\bin" "GO_DISABLE_OUTBOUND_NETWORK=false"] in dir C:\workdir\src
Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, windows/arm64.
Building packages and commands for target, windows/arm.
##### Testing packages.
exit status 0xc000007b
FAIL archive/tar 0.042s
exit status 0xc000007b
FAIL archive/zip 0.047s
exit status 0xc000007b
FAIL bufio 0.064s
exit status 0xc000007b
FAIL bytes 0.064s
exit status 0xc000007b
FAIL compress/bzip2 0.068s
exit status 0xc000007b
FAIL compress/flate 0.075s
exit status 0xc000007b
Comment From: gopherbot
Change https://golang.org/cl/345330 mentions this issue: dashboard: remove windows-arm-10 builder
Comment From: bcmills
Do we have any actual ARM hardware that we could copy the image to in order to determine whether the problem is with the WIndows version or the emulated host?
Comment From: zx2c4
We're running into a 0xc000007b failure when attempting to run arm32 binaries in our arm64 windows installation. It could be either the version of windows we have installed,
The latest insider build has arm32 issues, fyi, so be sure to run ordinary stable.
Things seem fine on real hardware.
Comment From: toothrot
Unfortunately, I couldn't get the stable version to run on the VM instances we're using. As I mentioned on #49266, I'll try updating the builders in my free time this weekend.
I also couldn't reproduce on real hardware, but this seems related to the specific environment we're running on.
Do you know if Windows ARM32 will continue to be supported?
Comment From: zx2c4
The idea is to run the ARM64 build of Windows and test ARM32 inside of it. Microsoft still supports ARM32 for IoT. If you think there's a good argument for sunsetting ARM32 in Go, though, then I guess that'd be a proposal to make, and then we could get some definition on it that way.
Comment From: toothrot
We will be giving this another shot, as we're working on upgrading our windows builders this week. The holiday will slow things down a bit, but we're actually working on it.
As a bonus, @heschi is doing it, which means updates won't be blocked solely on my availability.
Comment From: zx2c4
In case it helps,
PS C:\go-src\src> $env:GOARCH='arm'
PS C:\go-src\src> $env:GOHOSTARCH='arm'
PS C:\go-src\src> $env:GOARM='7'
PS C:\go-src\src> $env:GOOS='windows'
PS C:\go-src\src> $env:GOROOT_BOOTSTRAP='C:\go-binary-arm64-release'
PS C:\go-src\src> .\all.bat
You'll need to download llvm and such for cgo, but the above should hopefully get you past the, "it doesn't work at all!" phase, as the above does in fact work, and you can see ARM32 processes in the details view of task manager.
Comment From: zx2c4
@dmitshur Can we make this issue a goal for before 1.18 beta1? That way we're certain to have the beta1 period to fix bugs that might have slipped in since we went builderless a long while back, and therefore don't regress for 1.18.0.
Comment From: heschi
I'm actively working in this area, but it's a very complicated situation and I'm not confident it'll be done before beta.
Comment From: zx2c4
Anything I can do to help? What sorts of complications are you facing?
Comment From: heschi
The goal is to do this as part of #48946, but that's proving difficult.
If you were going to work some sort of miracle, getting Windows 11 working on an AWS ARM machine would be my preference.
Comment From: zx2c4
You might have luck running it under QEMU inside of Linux, on an AWS ARM instance. And then capping the number of CPUs to 4 or 8 or something smallish.
Comment From: heschi
Yeah, we've had some minimal success with that, but with anything more than 4 cores it greenscreens. Even with 4 I think we've seen some instability. See https://cs.opensource.google/go/x/build/+/master:env/windows-arm64/aws/. I just finished verifying that the situation isn't any better with a recent Windows 11 and qemu.
Comment From: zx2c4
Huh, I wonder if that instability still holds on the Windows 11 stable channel images.
Comment From: heschi
I was trying with 22504; given that we've seen the same failures with a 6-month build and a fresh one, I don't think there's much hope for stable to be better :(
Comment From: gopherbot
Change https://golang.org/cl/377696 mentions this issue: all: add Windows 11 builder
Comment From: heschi
@zx2c4 I finally, finally got to work on this and immediately faceplanted. Running anything with GOARCH=arm gives me This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.
We're running the latest version of Windows 11, not IOT. I can't find a clear answer to whether that's supposed to support ARM32 binaries. Do you know what's going on here?
Comment From: gopherbot
Change https://go.dev/cl/391136 mentions this issue: dashboard: update expectations for windows-arm64 builders
Comment From: heschi
@zx2c4 any ideas?
Comment From: gopherbot
Change https://go.dev/cl/460295 mentions this issue: all: remove macstadium integration and monitoring
Comment From: dmitshur
This issue was tracking adding this builder to the coordinator, but that's replaced with LUCI now. Closing this in favor of #67308.