Golang x/build: *-linux-arm builders have no disk space left

#!watchflakes
post <- builder ~ `-linux-arm$` && (`(N|n)o space left on device` || log ~ `(N|n)o space left on device`)

Issue created automatically to collect these failures.

Example (log):

FAIL    cmd/nm [build failed]
cmd/nm: mkdir /home/swarming/.swarming/w/ir/x/t/go-build1741138042/b3589/: no space left on device

— watchflakes

Comment From: gopherbot

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "cmd/nm" && test == ""

2025-06-13 16:04 gotip-linux-arm go@683810a3 cmd/nm (log)

FAIL cmd/nm [build failed] cmd/nm: mkdir /home/swarming/.swarming/w/ir/x/t/go-build1741138042/b3589/: no space left on device

2025-06-13 16:06 gotip-linux-arm go@64490589 cmd/nm (log)

FAIL cmd/nm [build failed] cmd/nm: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2227083791/b3589/: no space left on device

2025-06-13 16:09 gotip-linux-arm go@96a6e147 cmd/nm (log)

FAIL cmd/nm [build failed] cmd/nm: mkdir /home/swarming/.swarming/w/ir/x/t/go-build1630841084/b3589/: no space left on device

— watchflakes

Comment From: gopherbot

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "gotip-linux-arm" && (`no space left on device` || log ~ `no space left on device`)

2025-06-13 06:27 gotip-linux-arm go@91498761 cmd/link/internal/ld.TestSubprogramDeclFileLine/normal (log)

=== RUN TestSubprogramDeclFileLine/normal === PAUSE TestSubprogramDeclFileLine/normal === CONT TestSubprogramDeclFileLine/normal dwarf_test.go:119: ## build output: command-line-arguments: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3167359212/b001/: no space left on device dwarf_test.go:122: build error: exit status 1 --- FAIL: TestSubprogramDeclFileLine/normal (0.24s)

2025-06-13 06:27 gotip-linux-arm go@91498761 cmd/vendor/golang.org/x/tools/go/analysis/internal/analysisflags (log)

cmd/vendor/golang.org/x/tools/go/analysis/internal/analysisflags: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3471536847/b3691/: no space left on device

2025-06-13 06:27 gotip-linux-arm go@91498761 cmd/vendor/golang.org/x/tools/go/analysis/passes/errorsas (log)

cmd/vendor/golang.org/x/tools/go/analysis/passes/errorsas: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3471536847/b3660/: no space left on device

— watchflakes

Comment From: cherrymui

cc @golang/release

Comment From: dmitshur

I terminated the bot via swarming UI:

After it came back, it had a successful run on tip. So this might be fixed.

I took a look at the machine, it seems the container was running okay without running out of space for the last 16 months, until now. It has a 20 GB disk, which was 75% full before the container restarted (meaning 25% wasn't enough for a build), and now it's idling at 38%. There may be more to do if it keeps growing quickly. Closing for now, watchflakes should reopen if there's more to do.

Comment From: gopherbot

Found new dashboard test flakes for:

#!watchflakes
post <- builder == "gotip-linux-arm" && (`no space left on device` || log ~ `no space left on device`)

2025-07-09 15:28 gotip-linux-arm go@67c17044 cmd/go/internal/work.TestRespectSetgidDir (log)

=== RUN TestRespectSetgidDir build_test.go:230: TempDir: mkdir /home/swarming/.swarming/w/ir/x/t/TestRespectSetgidDir1379880301: no space left on device --- FAIL: TestRespectSetgidDir (0.00s)

2025-07-09 15:28 gotip-linux-arm go@67c17044 cmd/internal/archive.TestParseCGOArchive (log)

=== RUN TestParseCGOArchive archive_test.go:242: mkdir /home/swarming/.swarming/w/ir/x/t/TestGoobj1708184658: no space left on device --- FAIL: TestParseCGOArchive (0.01s)

2025-07-09 20:34 gotip-linux-arm go@8131635e cmd/compile.TestScript/dwarf5_gen_assembly_and_go (log)

=== RUN TestScript/dwarf5_gen_assembly_and_go === PAUSE TestScript/dwarf5_gen_assembly_and_go === CONT TestScript/dwarf5_gen_assembly_and_go run.go:223: 2025-07-09T21:49:48Z run.go:225: $WORK=/home/swarming/.swarming/w/ir/x/t/TestScriptdwarf5_gen_assembly_and_go1814236033/001 run.go:232: BOTO_CONFIG=/home/swarming/.swarming/w/ir/x/a/gsutil-bbagent/.boto CIPD_ARCHITECTURE=armv6l CIPD_CACHE_DIR=/home/swarming/.swarming/w/ir/cache/cipd_cache CIPD_PROTOCOL=v2 ... WORK=/home/swarming/.swarming/w/ir/x/t/TestScriptdwarf5_gen_assembly_and_go1814236033/001 TMPDIR=/home/swarming/.swarming/w/ir/x/t/TestScriptdwarf5_gen_assembly_and_go1814236033/001/tmp # Regression test case for bug #72810. Uses a build with # Go source files and assembly source files. (7.652s) > go build [stderr] uses.asm: mkdir /home/swarming/.swarming/w/ir/x/t/TestScriptdwarf5_gen_assembly_and_go1814236033/001/tmp/go-build3667244764/b001/: no space left on device run.go:232: FAIL: testdata/script/dwarf5_gen_assembly_and_go.txt:5: go build: exit status 1 --- FAIL: TestScript/dwarf5_gen_assembly_and_go (7.66s)

2025-07-09 20:34 gotip-linux-arm go@8131635e cmd/compile/internal/amd64 (log)

FAIL cmd/compile/internal/amd64 [build failed] cmd/compile/internal/compare: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2408519523/b3098/: no space left on device

2025-07-09 20:34 gotip-linux-arm go@8131635e cmd/compile/internal/arm (log)

FAIL cmd/compile/internal/arm [build failed] cmd/compile/internal/compare: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2408519523/b3098/: no space left on device

2025-07-10 16:27 gotip-linux-arm go@63ec70d4 cmd/cgo/internal/testcarchive.TestSIGPROF (log)

=== RUN TestSIGPROF === PAUSE TestSIGPROF === CONT TestSIGPROF carchive_test.go:1047: [/home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -buildmode=c-archive -o libgo6.a ./libgo6] sort: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b068/: no space left on device bytes: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b039/: no space left on device strings: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b044/: no space left on device hash: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b071/: no space left on device runtime/cgo: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b074/: no space left on device encoding/binary: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b069/: no space left on device text/tabwriter: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3413260242/b073/: no space left on device carchive_test.go:1049: exit status 1 --- FAIL: TestSIGPROF (0.33s)

2025-07-10 16:27 gotip-linux-arm go@63ec70d4 cmd/cgo/internal/testplugin.TestIssue18584 (log)

=== RUN TestIssue18584 ( GOPATH=$TMPDIR PWD=$TMPDIR/src/testplugin LD_LIBRARY_PATH=$TMPDIR/src/testplugin /home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -gcflags '' -buildmode=plugin -o plugin.so ./issue18584/plugin.go ) ( GOPATH=$TMPDIR PWD=$TMPDIR/src/testplugin LD_LIBRARY_PATH=$TMPDIR/src/testplugin /home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -gcflags '' -o issue18584.exe ./issue18584/main.go ) plugin_test.go:263: `/home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -gcflags -o issue18584.exe ./issue18584/main.go`: exit status 1 go: creating work dir: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2648728336: no space left on device --- FAIL: TestIssue18584 (1.06s)

2025-07-10 16:27 gotip-linux-arm go@63ec70d4 cmd/cgo/internal/testshared.TestGopathShlib (log)

=== RUN TestGopathShlib shared_test.go:584: executing `/home/swarming/.swarming/w/ir/x/w/goroot/bin/go install -buildmode=shared -linkshared ./depBase` failed exit status 1: internal/runtime/sys: go install internal/runtime/sys: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3986080036/b030/: no space left on device testshared/depBaseInternal: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3986080036/b060/: no space left on device unicode/utf8: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3986080036/b042/: no space left on device unicode: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3986080036/b059/: no space left on device --- FAIL: TestGopathShlib (0.26s)

2025-07-10 20:16 gotip-linux-arm go@aa5de9eb [build] (log)

Use of LUCI is subject to the Google [Terms of Service](https://policies.google.com/terms) and [Privacy Policy](https://policies.google.com/privacy) [I2025-07-10T20:57:41.406433Z 258141 0 sink.go:350] SinkServer: warm-up started [I2025-07-10T20:57:41.406596Z 258141 0 sink.go:423] SinkServer: starting HTTP server... [I2025-07-10T20:57:41.407562Z 258141 0 sink.go:355] SinkServer: warm-up ended [I2025-07-10T20:57:41.407856Z 258141 0 cmd_stream.go:569] rdb-stream: starting the test command - ["/home/swarming/.swarming/w/ir/cache/tools/bin/result_adapter" "go" "-v=false" "-dump-json" "/home/swarming/.swarming/w/ir/x/w/dist.testjson" "--" "/home/swarming/.swarming/w/ir/x/w/goroot/bin/go" "tool" "dist" "test" "-json"] go tool dist: Failed: exit status 1 ok archive/tar 0.233s ok archive/zip 0.522s ok bufio 0.084s ... --- FAIL: TestFakeTime (0.01s) crash_test.go:186: running /home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -o /home/swarming/.swarming/w/ir/x/t/go-build1175309002/testfaketime_-tags=faketime.exe -tags=faketime time_test.go:32: building testfaketime [-tags=faketime]: exit status 1 go: creating work dir: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2483299364: no space left on device --- FAIL: TestSUID (0.01s) crash_test.go:186: running /home/swarming/.swarming/w/ir/x/w/goroot/bin/go build -o /home/swarming/.swarming/w/ir/x/t/go-build1175309002/testsuid.exe security_test.go:84: building testsuid []: exit status 1 go: creating work dir: mkdir /home/swarming/.swarming/w/ir/x/t/go-build3868297829: no space left on device ... [W2025-07-10T21:02:54.954304Z 258141 0 cmd_stream.go:581] rdb-stream: test process exited with error: exit status 123 [I2025-07-10T21:02:54.954357Z 258141 0 cmd_stream.go:565] rdb-stream: the test process terminated [I2025-07-10T21:02:54.954524Z 258141 0 sink.go:448] SinkServer: shutdown started [I2025-07-10T21:02:54.954597Z 258141 0 sink.go:426] SinkServer: HTTP server stopped with "http: Server closed" [I2025-07-10T21:02:54.954617Z 258141 0 sink_server.go:96] SinkServer: draining TestResult channel started [I2025-07-10T21:02:54.954640Z 258141 0 sink_server.go:98] SinkServer: draining TestResult channel ended [I2025-07-10T21:02:54.954658Z 258141 0 sink_server.go:100] SinkServer: draining Artifact channel started [I2025-07-10T21:02:54.954679Z 258141 0 sink_server.go:102] SinkServer: draining Artifact channel ended [I2025-07-10T21:02:54.954748Z 258141 0 sink.go:451] SinkServer: shutdown completed successfully [I2025-07-10T21:02:54.954808Z 258141 0 cmd_stream.go:492] rdb-stream: exiting with 123

— watchflakes

Comment From: gopherbot

Sorry, but there were parse errors in the watch flakes script. The script I found was:

#!watchflakes
post <- builder ~ "linux-arm" && (`(N|n)o space left on device` || log ~ `(N|n)o space left on device`)

And the problems were:

script:2.19: ~ requires backquoted regexp

See https://go.dev/wiki/Watchflakes for details.

— watchflakes

Comment From: gopherbot

Found new dashboard test flakes for:

#!watchflakes
post <- builder ~ `-linux-arm$` && (`(N|n)o space left on device` || log ~ `(N|n)o space left on device`)

2025-07-10 16:49 x_tools-go1.23-linux-arm tools@4738c7c0 release-branch.go1.23@317a3c13 x/telemetry/counter/countertest (log)

golang.org/x/telemetry/counter/countertest: mkdir /home/swarming/.swarming/w/ir/x/t/go-build2276218715/b856/: no space left on device

2025-07-10 16:49 x_tools-go1.23-linux-arm tools@4738c7c0 release-branch.go1.23@317a3c13 x/tools/gopls/internal/cmd.TestWorkspaceSymbol (log)

=== RUN TestWorkspaceSymbol === PAUSE TestWorkspaceSymbol === CONT TestWorkspaceSymbol integration_test.go:1132: TempDir: mkdir /home/swarming/.swarming/w/ir/x/t/TestWorkspaceSymbol3982990736: no space left on device --- FAIL: TestWorkspaceSymbol (0.00s)

2025-07-10 16:49 x_tools-go1.23-linux-arm tools@4738c7c0 release-branch.go1.23@317a3c13 x/tools/gopls/internal/golang.TestUnify (log)

=== RUN TestUnify testfiles.go:87: go tool not available: /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.2.linux-arm/bin/go build -o /dev/null /home/swarming/.swarming/w/ir/x/t/testenv-267063516/main.go: exit status 1 command-line-arguments: mkdir /home/swarming/.swarming/w/ir/x/t/go-build716361053/b001/: no space left on device --- FAIL: TestUnify (0.16s)

2025-07-10 17:19 x_tools-go1.23-linux-arm tools@88a4eb30 release-branch.go1.23@317a3c13 x/tools/gopls/internal/test/integration/misc.TestRenamePackage_IntermediateTestVariant/default (log)

=== RUN TestRenamePackage_IntermediateTestVariant/default runner.go:179: mkdir /home/swarming/.swarming/w/ir/x/t/gopls-test-1956969006/TestRenamePackage_IntermediateTestVariant: no space left on device --- FAIL: TestRenamePackage_IntermediateTestVariant/default (0.00s)

2025-07-10 18:31 x_tools-go1.23-linux-arm tools@9563af69 release-branch.go1.23@317a3c13 x/tools/gopls/internal/test/integration/diagnostics.TestCreatingPackageInvalidatesDiagnostics_Issue66384/default (log)

=== RUN TestCreatingPackageInvalidatesDiagnostics_Issue66384/default runner.go:179: mkdir /home/swarming/.swarming/w/ir/x/t/gopls-test-2009159989/TestCreatingPackageInvalidatesDiagnostics_Issue66384: no space left on device --- FAIL: TestCreatingPackageInvalidatesDiagnostics_Issue66384/default (0.00s)

2025-07-10 18:31 x_tools-go1.23-linux-arm tools@9563af69 release-branch.go1.23@317a3c13 x/tools/gopls/internal/test/integration/workspace.TestChangeAddedWorkspaceFolders/add_an_already_added_file_but_with_an_ending_slash (log)

=== RUN TestChangeAddedWorkspaceFolders/add_an_already_added_file_but_with_an_ending_slash regtest.go:71: TempDir: mkdir /home/swarming/.swarming/w/ir/x/t/TestChangeAddedWorkspaceFoldersadd_an_already_added_file_but_with_an_ending_slash2373561732: no space left on device --- FAIL: TestChangeAddedWorkspaceFolders/add_an_already_added_file_but_with_an_ending_slash (0.00s)

2025-07-10 18:31 x_tools-go1.23-linux-arm tools@9563af69 release-branch.go1.23@317a3c13 x/tools/gopls/internal/test/marker.Test/token/range.txt (log)

=== RUN Test/token/range.txt === PAUSE Test/token/range.txt === CONT Test/token/range.txt marker_test.go:290: a.go:1:9-10 (token/range.txt:11:9-10): unexpected diagnostic: "initialization failed: packages.Load error: err: exit status 1: stderr: go: creating work dir: mkdir /home/swarming/.swarming/w/ir/x/t/go-build4201641618: no space left on device" marker_test.go:308: unmeetable expectation: once the following is met: completed work "diagnosing opened files" at least 1 time must have: no Error log messages reason: postcondition is not met: found matching log "2025/07/10 20:51:58 go/packages.Load #69: err: exit status 1: stderr: go: creating work dir: mkdir /home/swarming/.swarming/w/ir/x/t/go-build4201641618: no space left on device\n\n\tview_id=\"30\"\n\tsnapshot=0\n\tdirectory=/home/swarming/.swarming/w/ir/x/t/Testtokenrange.txt3589019529/001/work\n\tquery=[./ builtin]\n\tpackages=0\n\tduration=76.10128ms\n" --- FAIL: Test/token/range.txt (0.31s)

— watchflakes

Comment From: dmitshur

Closing since this hasn't happened for some time. It's been idling at around 40% disk usage, similar to its prior steady state in https://github.com/golang/go/issues/74169#issuecomment-2971994775.