Golang cmd/go: adding a new file by -overlay didn't work in Go 1.24, though this worked in Go 1.23

Go version

go version go1.24.0 windows/amd64

Output of `go env` in your module/workspace:

set AR=ar
set CC=gcc
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_ENABLED=1
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set CXX=g++
set GCCGO=gccgo
set GO111MODULE=
set GOAMD64=v1
set GOARCH=amd64
set GOAUTH=netrc
set GOBIN=
set GOCACHE=C:\Users\hajimehoshi\AppData\Local\go-build
set GOCACHEPROG=
set GODEBUG=
set GOENV=C:\Users\hajimehoshi\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFIPS140=off
set GOFLAGS=
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\HAJIME~1\AppData\Local\Temp\go-build1840174721=/tmp/go-build -gno-record-gcc-switches
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMOD=C:\Users\hajimehoshi\tmp\tmp2\go.mod
set GOMODCACHE=C:\Users\hajimehoshi\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\hajimehoshi\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\hajimehoshi\AppData\Roaming\go\telemetry
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.24.0
set GOWORK=
set PKG_CONFIG=pkg-config

What did you do?

Run this:

# Create a module to execute a test project (overlaytest).
# This is needed to get a local path for a module by `go list`.
mkdir tmp
cd tmp
go mod init foo
go get github.com/hajimehoshi/overlaytest@e210b75f

# Run overlaytest, which outputs 'Original' and generates overlay.json
go run github.com/hajimehoshi/overlaytest

# Run overlaytest with -overlay=overlay.json, which should replace the message from 'Original' to 'Replaced'
go run "-overlay=overlay.json" github.com/hajimehoshi/overlaytest

github.com/hajimehoshi/overlaytest does these things:

Runs a function Foo defined in internal/foo/foo.go. This function just outputs 'Original' by default.
Runs go list -f {{.Dir}} github.com/hajimehoshi/overlaytest/internal/foo to get the local path for the package.
Generates a JSON file to add a new file internal/foo/bar.go to replace the output message to 'Replaced'.

What did you see happen?

Both output 'Original'.

PS C:\Users\hajimehoshi\tmp> go run github.com/hajimehoshi/overlaytest
Original
PS C:\Users\hajimehoshi\tmp> go run "-overlay=overlay.json" github.com/hajimehoshi/overlaytest
Original

What did you expect to see?

First go-run should output 'Original' and the second should output 'Replaced'

Go 1.23.6 worked as expected (note that you might have to modify go.mod's Go version).

PS C:\Users\hajimehoshi\tmp> go1.23.6 run github.com/hajimehoshi/overlaytest
Original
PS C:\Users\hajimehoshi\tmp> go1.23.6 run "-overlay=overlay.json" github.com/hajimehoshi/overlaytest
Replaced

Comment From: hajimehoshi

On macOS and Linux, both Go 1.24 and 1.23 didn't work as expected:

% go version
go version go1.24.0 darwin/arm64
% go run github.com/hajimehoshi/overlaytest 
Original
% cat overlay.json
{"Replace":{"/Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/var/folders/cj/73zbb35j0qx5t4b6rnqq0__h0000gn/T/bar.go1091616997"}}
% go run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

% go1.23.6 run github.com/hajimehoshi/overlaytest 
Original
% cat overlay.json
{"Replace":{"/Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/var/folders/cj/73zbb35j0qx5t4b6rnqq0__h0000gn/T/bar.go4028295237"}}
% go1.23.6 run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

$ go version
go version go1.24.0 linux/arm64
$ go run github.com/hajimehoshi/overlaytest
Original
$ cat overlay.json
{"Replace":{"/home/parallels/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/tmp/bar.go2784474435"}}
$ go run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

$ go1.23.6 github.com/hajimehoshi/overlaytest
Original
$ cat overlay.json
{"Replace":{"/home/parallels/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/tmp/bar.go1458768758"}}
$ go1.23.6 run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

EDIT: My current guess is that replacing files in module cache might cause some troubles.

Comment From: hajimehoshi

CC @matloob per https://github.com/golang/go/issues/71075#issuecomment-2584437857

Comment From: hajimehoshi

ping @matloob ?

Comment From: matloob

(We had a US Holiday yesterday)

Yes, we don't respect overlays in the modcache. We should explicitly reject those cases.

I'll send a CL to explicitly reject them. But that CL wouldn't meet the bar to be backported, so it would show up in Go 1.25.

Comment From: hajimehoshi

Thank you for replying!

Yes, we don't respect overlays in the modcache. We should explicitly reject those cases.

1) Is there a way to use -overlay for external modules? 2) Is it guaranteed that -overlay always works for the standard libraries?

Comment From: jakebailey

It was suggested in https://github.com/golang/go/issues/69887#issuecomment-2560499446 that -overlay be used to modify code being built, e.g. for instrumenting code, in place of -toolexec. Blocking -overlay in dependencies seems like it'd greatly limit that approach (and potentially make it hard/impossible to use this method on projects split into multiple modules).

That and, -overlay powers gopls, so typing in the module cache (accidentally!) will probably lead to some strange effects when the open file overlay is used, or crashes if the go command bails out, right?

Comment From: hajimehoshi

Just FYI, my current -overlay usages are

to separate code under NDA for private repositories (e.g. Ebitengine's implementation for consoles)
to modify the standard library for consoles (https://ebitengine.org/en/blog/native_compiling_for_nintendo_switch.html)

Comment From: matloob

@hajimehoshi To answer the easier question first, yes -overlay works for the standard library. That shouldn't change.

On the other hand the module cache is intended to be read-only (we write the files read-only on disk) so the Go command assumes that it isn't modified, even by an overlay.

@jakebailey I think we would still be able to find a solution even if we don't use overlays. For instance, we might be able to use replace directives. We would plan to have gopls filter out the files provided to the overlay to avoid a crash. See https://github.com/golang/go/issues/71075#issuecomment-2666200658

@hajimehoshi I think replace directives are the best option for replacing code that's in the module cache. Would they work for your use cases? For the ebitengine NDA use case, could you replace the module containing the NDA'd code when doing the build in an environment where the implementation is available? You could perhaps split the specific interfaces into their own module and replace only that module to avoid having to replace a big module.

If it becomes necessary we can reverse course on this: for example, we could disable the module index when an overlay modifies the module cache. That should fix the immediate problems. But it may create other ones if we start breaking the assumption that the module index is immutable, so I think we should avoid that option if we can.

Comment From: hajimehoshi

@matloob Thanks! I'm relieved that -overlay for the standard library is fine!

I think replace directives are the best option for replacing code that's in the module cache. Would they work for your use cases? For the ebitengine NDA use case, could you replace the module containing the NDA'd code when doing the build in an environment where the implementation is available?

It is possible, but this means that a user has to modify their go.mod for their applications. Perhaps, I can develop a tool to do it automatically as a wrapper of a Go compiler, so this might work. It'd be like gomobile/gobind, which creates go.mod in a temporary directory, creates necessary sources there automatically. I'll try, thanks!

You could perhaps split the specific interfaces into their own module and replace only that module to avoid having to replace a big module.

This might work, but I don't want to expose the internal details as an exposed interface. Another concern is that the interface content can be changed anytime.

If it becomes necessary we can reverse course on this: for example, we could disable the module index when an overlay modifies the module cache. That should fix the immediate problems. But it may create other ones if we start breaking the assumption that the module index is immutable, so I think we should avoid that option if we can.

That makes sense.

Alright, I am convinced that -overlay for Go 1.23 on Windows worked for external modules but this was just a coincidence. Thank you for your insights!

Comment From: gopherbot

Change https://go.dev/cl/650475 mentions this issue: cmd/go: explicitly reject overlays affecting GOMODCACHE

Comment From: hajimehoshi

I've created my own helper library to overwrite files in external libraries: https://github.com/hajimehoshi/uwagaki

Comment From: erikb495

I had a very similar issue and a go clean -cache fixed it. I don't know why. golang can't find SOME standard library packages (could not import fmt (open : no such file or directory))](https://stackoverflow.com/questions/79715822/golang-cant-find-some-standard-library-packages-could-not-import-fmt-open-nl)

Golang cmd/go: adding a new file by -overlay didn't work in Go 1.24, though this worked in Go 1.23

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Output of `go env` in your module/workspace: