The tests under golang.org/x/tools almost always fail on the linux-arm builder.

Looking at the build dashboard, the most frequent failure mode (by far) seems to result in errors of the form fork/exec /workdir/go/bin/go: cannot allocate memory *os.PathError in the golang.org/x/tools/internal/lsp tests:

https://build.golang.org/log/5fb7985054d8b682b81438c385ff16e8325a3aa6 https://build.golang.org/log/e422caa59d35f5c76e097fdadb60407ee2807368 https://build.golang.org/log/3e3e538cf33e70880da37c2dd1c716b733b4b009 https://build.golang.org/log/739f069936cc47cc800b4d898c505e17a9d4a7e1

However, some of the other tests have also flaked with various resource-exhaustion errors:

A similar cannot allocate memory failure mode for golang.org/x/tools/go/internal/gcimporter: https://build.golang.org/log/99a21d9e2e1064d08890439b43de987e0bfcdf52

fatal error: runtime: out of memory in golang.org/x/tools/go/ssa: https://build.golang.org/log/de56b0ffe0e39b38b9ed449bd0cd382be72eda69

pthread_create failed: Resource temporarily unavailable and no space left on device: https://build.golang.org/log/7151dca16e38a6f61051f032b2f6b713253052ac

I don't know exactly what the resource footprint of the linux-arm builder is (@andybons or @dmitshur might know?), but if the x/tools tests are expected to pass within that footprint we may need to dial back the test concurrency on that builder.

CC @ianthehat

Comment From: dmitshur

I don't know exactly what the resource footprint of the linux-arm builder is

Our linux/arm builders are on Scaleway, see here. There are 50 builders and they're C1 BareMetal instances. Documentation at https://www.scaleway.com/en/bare-metal-instances/ says C1 instances have 2 GB of RAM, 4 ARMv7 cores.

2 GB of RAM certainly isn't a lot. The next step up, C2S, has 8 GB RAM, but also 64-bit CPUs instead of the 32-bit we need (not sure if that's a blocker or not).

Comment From: gopherbot

Change https://golang.org/cl/192332 mentions this issue: dashboard: skip x/tools on the linux-arm builder

Comment From: bradfitz

The other possibility is to use go test -p 1 on the linux-arm builder.

That'd require a bit of coordinator changes.

Comment From: bcmills

I'm not entirely sure that would even fix it: I don't know how much of the memory usage is due to parallelism vs. large test-cases, or how much of it ends up in long-running LSP servers.

Comment From: gopherbot

Change https://golang.org/cl/192336 mentions this issue: go/packages,internal/lsp: skip memory-intensive tests on the linux-arm builder

Comment From: gopherbot

Change https://golang.org/cl/192578 mentions this issue: all: skip more memory-intensive tests on linux-arm

Comment From: millerresearch

This issue applies (even more so) to the plan9-arm builders, which are mainly Raspberry Pi 3B+ boards with 1GB of RAM. I'll submit a CL to make the ExitIfSmallMachine test apply to plan9-arm, as it does to linux-arm.

Comment From: gopherbot

Change https://golang.org/cl/291469 mentions this issue: gopls/internal/regtest: exit if small machine

Comment From: gopherbot

Change https://golang.org/cl/322409 mentions this issue: internal/testenv: update linux-arm to linux-arm-scaleway

Comment From: dmitshur

The Scaleway linux/arm builders are no more, and there haven't been recent reports here. If this comes up on the newer linux/arm builders, watchflakes will file a report. Closing.