Golang cmd/go: go tool -n can point to a file that doesn't exist

Go version

go version go1.24.1 darwin/arm64

Output of `go env` in your module/workspace:

AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/ianwahbe/go/bin'
GOCACHE='/Users/ianwahbe/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/ianwahbe/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build3052038314=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/ianwahbe/go/src/github.com/example/empty/go.mod'
GOMODCACHE='/Users/ianwahbe/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ianwahbe/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.24.1/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/ianwahbe/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.24.1/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I was using go tool -n github.com/iwahbe/helpmakego in my Makefile to work around #71733 like this:

HELPMAKEGO := $(shell go tool -n github.com/iwahbe/helpmakego)

# Use the tool as intended - details don't really matter here
$(shell ${HELPMAKEGO} ...)

I noticed that sometimes $(shell ${HELPMAKEGO} ...) would fail unexpectedly, saying that there was no file at ${HELPMAKEGO}.

I can replicate this behavior locally:

$ mkdir repro
$ go mod init
$ go get -tool github.com/iwahbe/helpmakego
$ go tool -n github.com/iwahbe/helpmakego | tee should_exist && rm $(cat should_exist)
/Users/ianwahbe/Library/Caches/go-build/c5/c503c5059f1c48a29b1dc68749074f475aaf82c717ef355e7079f7e9b4116c2b-d/helpmakego
$ go tool -n github.com/iwahbe/helpmakego | tee should_exist && rm $(cat should_exist)
/var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build1160981703/b001/exe/helpmakego
rm: /var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build1160981703/b001/exe/helpmakego: No such file or directory
$ go tool -n github.com/iwahbe/helpmakego | tee should_exist && rm $(cat should_exist)
/Users/ianwahbe/Library/Caches/go-build/c5/c503c5059f1c48a29b1dc68749074f475aaf82c717ef355e7079f7e9b4116c2b-d/helpmakego
$ go tool -n github.com/iwahbe/helpmakego | tee should_exist && rm $(cat should_exist)
/var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build1221207145/b001/exe/helpmakego
rm: /var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build1221207145/b001/exe/helpmakego: No such file or directory

You will see that every other time go tool -n github.com/iwahbe/helpmakego it points to a file that doesn't exist.

What did you see happen?

$ go tool -n github.com/iwahbe/helpmakego
/var/folders/fg/_1q36r4j6yx0rwz2fbhjd5y40000gn/T/go-build1221207145/b001/exe/helpmakego

What did you expect to see?

I expect that any time go tool -n ... is run, it will point to a runnable executable.

If that isn't true, then go help tool should document what conditions are necessary to ensure that go tool -n ... will point to an executable file.

Comment From: gabyhelp

Related Issues

Related Code Changes

cmd/go: use os.Executable to find GOROOT

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: seankhliao

seems like when it builds a tool it prints the location of it in the temporary work directory, rather than the final location in the build cache

Comment From: ianlancetaylor

CC @matloob @samthanawalla

Comment From: KirtanSoni

I figured out the cause on the bug. would u mind if i take this issue up and propose a PR

Comment From: gopherbot

Change https://go.dev/cl/658395 mentions this issue: cmd/go: respect -n flag in 'go tool' command

Comment From: matloob

The behavior in your repro log doesn't match what I'd expect: specifically my expectation is that we return the temp file location the first time we run go tool (where the tool is not yet cached) and the location to the cached path afterwards. Here's a log from my repro:

matloob@matloob-mac Desktop % mkdir repro
matloob@matloob-mac Desktop % go mod init example.com/repro
go: creating new go.mod: module example.com/repro
go: to add module requirements and sums:
    go mod tidy
matloob@matloob-mac Desktop % go get -tool github.com/iwahbe/helpmakego
go: downloading github.com/iwahbe/helpmakego v0.1.0
go: added github.com/inconshreveable/mousetrap v1.1.0
go: added github.com/iwahbe/helpmakego v0.1.0
go: added github.com/spf13/cobra v1.8.1
go: added github.com/spf13/pflag v1.0.5
go: added golang.org/x/mod v0.22.0
matloob@matloob-mac Desktop % go tool -n github.com/iwahbe/helpmakego
/var/folders/k3/3s_zz9nx1ms089s9vjb70xw8004_sl/T/go-build1617190989/b001/exe/helpmakego

matloob@matloob-mac Desktop % 
matloob@matloob-mac Desktop % go tool -n github.com/iwahbe/helpmakego
/Users/matloob/Library/Caches/go-build/11/1102170b7a6b4ae7d914933cb74aac17d22032c6b28f695dfdafa976774e1f5c-d/helpmakego
matloob@matloob-mac Desktop % rm /Users/matloob/Library/Caches/go-build/11/1102170b7a6b4ae7d914933cb74aac17d22032c6b28f695dfdafa976774e1f5c-d/helpmakego

matloob@matloob-mac Desktop % go tool -n github.com/iwahbe/helpmakego
/var/folders/k3/3s_zz9nx1ms089s9vjb70xw8004_sl/T/go-build3727914114/b001/exe/helpmakego
matloob@matloob-mac Desktop % /var/folders/k3/3s_zz9nx1ms089s9vjb70xw8004_sl/T/go-build3727914114/b001/exe/helpmakego
zsh: no such file or directory: /var/folders/k3/3s_zz9nx1ms089s9vjb70xw8004_sl/T/go-build3727914114/b001/exe/helpmakego

What's happening is that the first time we build the binary, we build the executable to a temporary directory and then execute the command from the built location. (This is what go run did before executable caching was added) Every subsequent time, we fetch the executable from the cache and run it from there. If we decide to, we can make the example work by setting the 'built' field to be the cache location of the executable in the last condition in cmd/go/internal/work.(*Builder).updateBuildID.

I'm a bit unsure of doing this because the cache is supposed to be an implementation detail of the go command and it might not be a good idea for tools to depend on the path of the file in the cache. It is better to run go build -o to fetch the binary from the cache to a location controlled by the user, and then run the binary from there.

Of course if we decide that the file returned by -n isn't always available, we should update the documentation to indicate that the tool may not necessarily be available from the location returned by go tool -n.

Comment From: KirtanSoni

@matloob I wanted to propose the same change considering a different implementation (behind the -n flag), to point the fresh cache to a.built, and poison the target. Another workaround I can propose is that we can print out an error if the cache misses, as it will be futile to try and run the build that doesn't exist. This way we can isolate the implementation in go tool.

Also in my opinion, with the -n flag, that output is not usable when run without a cache. I would propose to keep the implementation of -n similar to the other go commands, which would make it consistent and easier to work with.

Comment From: WGH-

This issue prevents me from using it in protoc like this:

protoc --plugin=protoc-gen-go="$(go tool -n google.golang.org/protobuf/cmd/protoc-gen-go)"

Comment From: joeycumines

Seems you can invoke go tool -n <pkg> twice, and use the second output.

Is this issue perhaps related to the stability of the output path?

Here is a simple replicator:

go get -tool golang.org/x/tools/cmd/godoc

while :; do
  a="$(go tool -n golang.org/x/tools/cmd/godoc)" &&
    b="$(go tool -n golang.org/x/tools/cmd/godoc)" &&
    c="$(go tool -n golang.org/x/tools/cmd/godoc)" &&
    printf 'First: %s\nSecond: %s\nThird: %s\n' "$a" "$b" "$c" &&
    if [ "$a" != "$b" ]; then echo "Output changed on the second attempt!"; fi &&
    if [ "$b" != "$c" ]; then echo "Output changed on the third attempt!"; fi &&
    rm "$b" ||
    echo "Failed to remove the second file!"
done

As a baseline I also tried go tool -n followed by sleep 5, which did not resolve the issue.

Comment From: mvdan

I ran into this as well (https://github.com/golang/go/issues/74650).

It is better to run go build -o to fetch the binary from the cache to a location controlled by the user, and then run the binary from there.

I disagree with this conclusion. Often, the user does not have a good location to place and cache the binary. Here are some reasonable options that come to mind: * The current directory; it gets in the way, especially when working inside a VCS repository, which adds an untracked file * A temporary directory, such as under /tmp; it causes the binary to disappear too often, e.g. at every reboot * A user cache directory, like ~/.cache; it's not easy to do in a portable way (os.UserCacheDir exists for a reason), one must choose a directory name carefully, and the files may be left behind practically forever without any trim/cleanup process

$GOCACHE resolves all of these problems. One of the main selling points of https://github.com/golang/go/issues/48429 was that the tool binaries would be cached, so I think it's perfectly reasonable for go tool -n to give us the location inside the cache.

I'm a bit unsure of doing this because the cache is supposed to be an implementation detail of the go command and it might not be a good idea for tools to depend on the path of the file in the cache.

I agree that the layout and properties of $GOCACHE are an implementation detail, but "give me a path to the cached binary, which I can expect to exist for at least the next hour" is not at odds with that, in my opinion. You can alter the way that $GOCACHE works in many ways without breaking this user feature. And I think the feature is really useful and should be supported.