Go version

go1.25-devel_cfb4e9bc4a

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE='on'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/rhang/gocode/bin'
GOCACHE='/Users/rhang/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/rhang/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/f6/p6sx9gt977l5n0hngdbjpnch0000gn/T/go-build3572606388=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/rhang/gocode/src/github.com/golang/go/src/go.mod'
GOMODCACHE='/Users/rhang/gocode/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/rhang/gocode'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/rhang/gocode/src/github.com/golang/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/rhang/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/rhang/gocode/src/github.com/golang/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25-devel_cfb4e9bc4a Tue Jun 3 13:34:49 2025 -0700'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

While testing upcoming go1.25 changes on our internal monorepo, we noticed this build failure for all go Bazel targets.

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
2025-06-09 00:44:46 PDT
compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/pack: fork/exec external/go_sdk/pkg/tool/linux_amd64/pack: no such file or directory
2025-06-09 00:44:46 PDT
ERROR: /home/testuser/.cache/bazel-single-base/external/io_nhooyr_websocket/BUILD.bazel:3:11: GoCompilePkg external/io_nhooyr_websocket/websocket.a failed: (Exit 1): builder failed: error executing GoCompilePkg command (from target @@io_nhooyr_websocket//:websocket) bazel-out/k8-opt-exec-ST-a828a81199fe/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -goroot bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/stdlib_ -installsuffix ... (remaining 63 arguments skipped)

What did you see happen?

While building go after https://github.com/golang/go/commit/cfb4e9bc4ae957dba63cb2ee5e020fcd25d553fd the pack command is missing.

../bin/go tool
asm
cgo
compile
cover
link
preprofile
vet

What did you expect to see?

Since rules_go for Bazel relies on the pack tool https://github.com/bazel-contrib/rules_go/blob/master/go/tools/builders/compilepkg.go#L504 to emulate the go toolchain compiler logic https://github.com/golang/go/blob/master/src/cmd/go/internal/work/exec.go#L965 would it be possible to keep the pack command exposed for lower-level callers of the go toolchain?

Comment From: gabyhelp

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: dmitshur

Thanks for the report. The corresponding release note for this is https://tip.golang.org/doc/go1.25#go-command (second paragraph, "The Go distribution will include fewer prebuilt tool binaries. …").

If rules_go were updated to use go tool pack to invoke the pack command, would that resolve the issue for you?

CC @golang/command-line.

Comment From: r-hang

Yes, I think that should work.

I can confirm that this patch to rules_go resolves the issue

diff --git a/go/private/actions/compilepkg.bzl b/go/private/actions/compilepkg.bzl
index 3b7f27ae..6b578fa7 100644
--- a/go/private/actions/compilepkg.bzl
+++ b/go/private/actions/compilepkg.bzl
@@ -202,6 +202,7 @@ def emit_compilepkg(
         inputs_direct.append(go.mode.pgoprofile)

     go.actions.run(
+        tools = [go.toolchain.sdk.go],
         inputs = depset(inputs_direct, transitive = inputs_transitive),
         outputs = outputs,
         mnemonic = "GoCompilePkgExternal" if is_external_pkg else "GoCompilePkg",
diff --git a/go/tools/builders/compilepkg.go b/go/tools/builders/compilepkg.go
index d9e6454f..f7250778 100644
--- a/go/tools/builders/compilepkg.go
+++ b/go/tools/builders/compilepkg.go
@@ -501,7 +501,7 @@ func compileGo(goenv *env, srcs []string, packagePath, importcfgPath, embedcfgPa

 func appendToArchive(goenv *env, outPath string, objFiles []string) error {
        // Use abs to work around long path issues on Windows.
-       args := goenv.goTool("pack", "r", abs(outPath))
+       args := goenv.goCmd("tool", "pack", "r", abs(outPath))
        args = append(args, objFiles...)
        return goenv.runCommand(args)
 }

I apologize for the confusion. I made a mistake before filing this issue, during my investigation, when I got to a state where my locally built go tool pack complained that the pack command was also not recognized. This is no longer the case.

Thanks, @dmitshur!

Comment From: r-hang

Hey @dmitshur, sorry for the false resolution report. I must have had some cached state when applying the patch to rules_go. If I apply the fix I mentioned above I get

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
go: no such tool "pack"
compilepkg: error running subcommand external/go_sdk/bin/go: exit status 2
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 91.164s, Critical Path: 28.20s
INFO: 266 processes: 51 disk cache hit, 174 internal, 41 processwrapper-sandbox.

while seems to be a variation of the original error message of

compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/pack: fork/exec external/go_sdk/pkg/tool/linux_amd64/pack: no such file or directory

Tracking the error down to https://cs.opensource.google/go/go/+/refs/tags/go1.25rc2:src/cmd/go/internal/base/tool.go;l=20 it seems like go tool pack looks at the pkg/tools directory for the command https://cs.opensource.google/go/go/+/refs/tags/go1.25rc2:src/go/build/gc.go;l=16 that gets removed in go1.25.

I tried to modify rules_go to build the pack command and copy over that to the pkg/tools directory which does work locally for me but it violates some expectations in rules_go tests. Here's a commit with that fix attempt in rules_go https://github.com/bazel-contrib/rules_go/commit/d677f95908fb9055035db125e2787fc42e8a232d

--- FAIL: Test (49.04s)
--- FAIL: Test/Check_Hashes (0.58s)
reproducibility_test.go:240: k8-fastbuild/bin/external/io_bazel_rules_go/stdlib_/pkg/tool/linux_amd64/pack is different: ccd942a974ab>
k8-fastbuild-ST-f56123538599/bin/external/io_bazel_rules_go/stdlib_/pkg/tool/linux_amd64/pack is different: 6e3aa488a0300d3ea0b3d>
--- FAIL: Test/Check_builder (0.03s)
reproducibility_test.go:264: Found go_sdk path in builder binary, builder tool won't be reproducible

I've opened a corresponding issue on rules_go https://github.com/bazel-contrib/rules_go/issues/4398

Comment From: r-hang

Any chance a resolution will be included in go1.25rc3?

Comment From: matloob

@r-hang go tool pack shouldn't need to look in pkg/tool: it will build and run the pack tool when there's no pack binary in pkg/tool.

Could you give me instructions to reproduce the error you see with the rules_go repo? I can take a look and see what's going on.

Comment From: r-hang

Hey @matloob

I just created a minimal repro in a small fork of the Bazel Go tutorial repo https://bazel.build/start/go at https://github.com/r-hang/examples

To reproduce

> git clone git@github.com:r-hang/examples.git
> cd examples/go-tutorial/stage3
> bazel build ... 
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
compilepkg: error running subcommand external/rules_go~~go_sdk~main___download_0/pkg/tool/linux_amd64/pack: fork/exec external/rules_go~~go_sdk~main___download_0/pkg/tool/linux_amd64/pack: no such file or directory
Use --verbose_failures to see the command lines of failed build steps.

To fix the issue, update examples/go-tutorial/stage3/MODULE.bazel file to use "1.24.5"

$ git diff

diff --git a/go-tutorial/stage3/MODULE.bazel b/go-tutorial/stage3/MODULE.bazel
index 75616af..acbd599 100644
--- a/go-tutorial/stage3/MODULE.bazel
+++ b/go-tutorial/stage3/MODULE.bazel
@@ -4,4 +4,4 @@ bazel_dep(
 )

 go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
-go_sdk.download(version = "1.25rc2") # use 1.24.5 to avoid the error
+go_sdk.download(version = "1.24.5") # use 1.24.5 to avoid the error

$  bazel build ...
INFO: Invocation ID: f8624ed4-0faa-44af-8eb0-949be652c880
INFO: Analyzed 3 targets (2 packages loaded, 4968 targets configured).
INFO: Found 3 targets...
INFO: Elapsed time: 28.579s, Critical Path: 24.11s
INFO: 16 processes: 3 internal, 13 processwrapper-sandbox.
INFO: Build completed successfully, 16 total actionsg

My changes on top of the Bazel Go tutorial starter project (https://github.com/r-hang/examples/pull/1/files) are to use go1.25rc2, use the rules_go version (0.56.1) that supports go1.25, and modify one of the dependencies to be a CGO dependency in order to have rules_go trigger a call to the "pack" command in the compilation step https://github.com/bazel-contrib/rules_go/blob/4957aa38634a54af69d1d02e510b0ec9df73619d/go/tools/builders/compilepkg.go#L459

Follow up on developing on rules_go to use 'go tool pack`

To use 'go tool pack' instead of the direct path to the pack command that was removed, we have to use a local copy of rules_go.

This can be done by getting a local copy of rules_go at 0.56.1

$ git clone git@github.com:bazel-contrib/rules_go.git
$ git fetch origin v0.56.0:v0.56.1 && git checkout v0.56.1

And adding a local override to go-tutorial/stage3/MODULE.bazel to the path of the local rules_go checkout

local_path_override(
    module_name = "rules_go",
    path = "/home/user/gocode/src/github.com/bazel-contrib/rules_go",
)

And then modifying the rules_go logic to use go tool pack instead of directly pathing to the removed pack command.

~/g/s/g/b/rules_go ❯❯❯ git diff
diff --git a/go/private/actions/compilepkg.bzl b/go/private/actions/compilepkg.bzl
index 73827d71..a68da51b 100644
--- a/go/private/actions/compilepkg.bzl
+++ b/go/private/actions/compilepkg.bzl
@@ -200,6 +200,7 @@ def emit_compilepkg(
     go.actions.run(
         inputs = depset(inputs_direct, transitive = inputs_transitive),
         outputs = outputs,
+        tools = [go.toolchain.sdk.go],
         mnemonic = "GoCompilePkgExternal" if is_external_pkg else "GoCompilePkg",
         executable = go.toolchain._builder,
         arguments = ["compilepkg", shared_args, compile_args],
diff --git a/go/tools/builders/compilepkg.go b/go/tools/builders/compilepkg.go
index 7c2d7049..f9f3e8f0 100644
--- a/go/tools/builders/compilepkg.go
+++ b/go/tools/builders/compilepkg.go
@@ -531,7 +531,8 @@ func compileGo(goenv *env, srcs []string, packagePath, importcfgPath, embedcfgPa

 func appendToArchive(goenv *env, outPath string, objFiles []string) error {
        // Use abs to work around long path issues on Windows.
-       args := goenv.goTool("pack", "r", abs(outPath))
+       // args := goenv.goTool("pack", "r", abs(outPath))
+       args := goenv.goCmd("tool", "pack", "r", abs(outPath))
        args = append(args, objFiles...)
        return goenv.runCommand(args)
 }

The error should now be

$ bazel build ...
INFO: Invocation ID: 4c0370f7-fe99-4841-8d34-327e45e1ce2f
INFO: Analyzed 3 targets (1 packages loaded, 3572 targets configured).
ERROR: /home/user/gocode/src/github.com/bazelbuild/examples/go-tutorial/stage3/fortune/BUILD:3:11: GoCompilePkg fortune/fortune.a failed: (Exit 1): builder failed: error executing GoCompilePkg command (from target //fortune:fortune) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_go~~go_sdk~main___download_0/builder_reset/builder compilepkg -sdk external/rules_go~~go_sdk~main___download_0 -goroot ... (remaining 37 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
go: no such tool "pack"
compilepkg: error running subcommand external/rules_go~~go_sdk~main___download_0/bin/go: exit status 2
ERROR: /home/user/gocode/src/github.com/bazelbuild/examples/go-tutorial/stage3/fortune/BUILD:11:8: GoCompilePkg fortune/fortune_test.internal.a failed: (Exit 1): builder failed: error executing GoCompilePkg command (from target //fortune:fortune_test) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_go~~go_sdk~main___download_0/builder_reset/builder compilepkg -sdk external/rules_go~~go_sdk~main___download_0 -goroot ... (remaining 41 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
go: no such tool "pack"
compilepkg: error running subcommand external/rules_go~~go_sdk~main___download_0/bin/go: exit status 2
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 27.594s, Critical Path: 24.67s
INFO: 9 processes: 5 internal, 4 processwrapper-sandbox.
ERROR: Build did NOT complete successfully

Debugging Note

Something I also tried in rules_go thinking that go toolchain was having difficulty boostrapping the pack command: I removed src/cmd/** from the go sdk Bazel sandbox materialized source files exclusion list https://github.com/bazel-contrib/rules_go/blob/4957aa38634a54af69d1d02e510b0ec9df73619d/go/private/BUILD.sdk.bazel#L31 but the erroring result is still the same.

Comment From: matloob

cc @jitsu-net

Comment From: matloob

@r-hang

Hi, I'm a bit rusty on my Bazel knowledge so I couldn't get something to work, but I think I understand what's going on and hopefully with your and the rules_go maintainers' help we can update rules_go.

The go command expects to be able to build the pack command from source when it's needed when running go tool pack. So we can either make the source for cmd/pack available to it from the sdk stdlib (I think bazel removes the cmd/ directory when it prepares the standard library in the toolchain rules?) or we can make a compiled pack tool available from the pkg/tool/$GOOS_$GOARCH the the go command uses to find precompiled tools. Is sounds like making the compiled tool available is probably the right choice since we already precompile the standard library.

So I think when we precompile the standard library in the sdk rules using "go install" to place it in the GOROOT we can add cmd/pack to the list of packages to install. And then make sure that the installed pack binary is included in the GOROOT pkg/tool directory used by the go command.

Maybe something along these lines?

diff --git a/go/private/actions/stdlib.bzl b/go/private/actions/stdlib.bzl
index 2ca42814..17f3fddf 100644
--- a/go/private/actions/stdlib.bzl
+++ b/go/private/actions/stdlib.bzl
@@ -140,6 +140,7 @@ def _build_stdlib(go):
     if go.mode.msan:
         args.add("-msan")
     args.add("-package", "std")
+    args.add("-package", "cmd/pack")
     if not go.mode.pure:
         args.add("-package", "runtime/cgo")

Which doesn't seem to work but I wonder if we need to do some more work to make sure the installed pack tool is available.

Comment From: r-hang

Hey, thanks for shedding more light on the problem

If go command isn't able to build pack command from source, do you have insights into why that might be the case? Would we expect the go command to throw an error message that source files were missing and pack couldn't be built and run?

As for making a compiled pack tool available via go install by adding -package cmd/pack. I'm probably running into the error that you might have seen in your investigation

# cmd/pprof
flag provided but not defined: -trimpath
usage: link [options] main.o

Seems like "std" is doable as the current setup works but "cmd" or "cmd/pack" isn't.

Looking at the docs for go install, however I see

- Arguments must be package paths or package patterns (with "..." wildcards). They must not be standard packages (like fmt), meta-patterns (std, cmd, all), or relative or absolute file paths.

so that seems like it's not expected to work as it's a meta pattern?

A few weeks ago, I experimented with building the pack command and copying it to the tools path but the error messages I got in the unit tests seems like I was violating some conventions of rules_go details are at https://github.com/bazel-contrib/rules_go/commit/d677f95908fb9055035db125e2787fc42e8a232d From this experience I was going in the direction of making source for cmd/pack available.

Any ideas @fmeum?

Comment From: matloob

@r-hang

If go command isn't able to build pack command from source, do you have insights into why that might be the case? Would we expect the go command to throw an error message that source files were missing and pack couldn't be built and run?

We're running into another weird case in go tool. Before trying to build and run a tool that doesn't exist in pkg/tool, it will double check that it's a tool that ships in the distribution my making sure the source directory for it exists in src/cmd. Because that directory is missing, it considers the tool to not be a standard library tool. And that's happening because we remove the cmd source code when we package up the standard library. (In the SDK's build.bazel we exclude src/cmd/*** from the srcs filegroup).

As for making a compiled pack tool available via go install by adding -package cmd/pack. I'm probably running into the error that you might have seen in your investigation

```

cmd/pprof

flag provided but not defined: -trimpath usage: link [options] main.o ```

Oh I wasn't able to get the changes to run locally, but it seems like the change may need to be a bit more complicated. I see that we add -trimpath to the -ldflags in go/tools/builders/stdlib.go. Maybe the flag is being passed to another binary, but the linker doesn't seem to have a -trimpath flag defined. The go command itself overrides the GOROOT passed to the linker when trimpath is passed to it.

On the commit you linked to, could you try running go install cmd/pack instead of go building it? The go install command should build and place cmd/pack in the pkg/tool/$GOOS_$GOARCH directory itself, and you wouldn't need to set all the other bits. What were the error messages you were getting? Your change is close to what I was thinking of.

Comment From: r-hang

An update here, with the insights from your comments on the issue I put up https://github.com/bazel-contrib/rules_go/pull/4420 which passes rules_go CI and is able to build all the targets in my company's pretty massive Go monorepo.

The code is much simpler with your suggestion of just using go install instead of go build

Comment From: dmitshur

It sounds like there are no changes planned in cmd/dist for Go 1.25, so closing. Thanks.