Go version

go version go1.21.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/trajan0x/Library/Caches/go-build'
GOENV='/Users/trajan0x/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/trajan0x/go/pkg/mod'
GONOPROXY='github.com/synapsecns/*'
GONOSUMDB='github.com/synapsecns/*'
GOOS='darwin'
GOPATH='/Users/trajan0x/go'
GOPRIVATE='github.com/synapsecns/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/12/8xtw48x951g0vv4z_ctcr2hr0000gn/T/go-build1424252971=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

  1. git clone https://github.com/synapsecns/sanguine
  2. go work sync
  3. git checkout 995354fca4e89c0931ea5f6f0a77f170aac2f13e (this is happening across a couple commits and I found it on synapsecns/sanguine#1840, just providing a single commit on master for reproducibility)

(Note: recipe won't work for this kinda bug, but I created a repl of the repo here and ran the command just for ease of debugging.

What did you see happen?

go: cloud.google.com/go@v0.110.7 requires
        github.com/google/s2a-go@v0.1.4
panic: internal error: found a version conflict, but no constraint it violates

goroutine 1 [running]:
cmd/go/internal/modload.editRequirements({0x1010edbd0, 0x1014830a0}, 0x14013bbaaa0, {0x0, 0x0, 0x1400bb7e828?}, {0x14007bac000, 0x108, 0x1400bb7e800?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/edit.go:336 +0x2bc0
cmd/go/internal/modload.EditBuildList({0x1010edbd0, 0x1014830a0}, {0x0, 0x0, 0x0}, {0x14007bac000, 0x108, 0x200})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/buildlist.go:640 +0x64
cmd/go/internal/workcmd.runSync({0x1010edbd0, 0x1014830a0}, 0x140000265b8?, {0x14?, 0x1010248c0?, 0x100adbc6c?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/workcmd/sync.go:117 +0x4d0
main.invoke(0x10143fd80, {0x140000201a0, 0x1, 0x1})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:268 +0x4f0
main.main()
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:186 +0x754

What did you expect to see?

Success, another error, but not a panic.

The fact that this isn't handled gracefully tells me it's probably unintended behavior. But I'm a little unsure based on the context in edit.go

Comment From: bcmills

Some notes: 1. This does appear to be perfectly reproducible in synapsecns/sanguine@995354fca4e89c0931ea5f6f0a77f170aac2f13e. 2. github.com/google/s2a-go@v0.1.4 is at go 1.16 and therefore unpruned. It is required directly by several of the modules in the workspace.

When a pruned module has a requirement on another root module, and that root is unpruned, we need to transition from the pruned side of the module graph to the unpruned side. I suspect that isn't happening when it needs to in dqTracker.path.

Comment From: gopherbot

Change https://go.dev/cl/571800 mentions this issue: cmd/go/internal/modload: follow dependencies of unpruned roots in dqTracker.path

Comment From: bcmills

The bug is triggering in the contrib/terraform-provider-helmproxy module, and the conflict is coming about because the workspace-selected version of github.com/google/pprof is ending up lower than the module-selected version — but github.com/google/pprof isn't relevant to this module anyway.

sanguine/contrib/terraform-provider-helmproxy$ go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210506205249-923b5ab0fc1a

sanguine/contrib/terraform-provider-helmproxy$ GOWORK=off go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1

Comment From: bcmills

The dependency on the mismatched pprof comes through an old k8s.io/client-go@v0.25.5:

k8s.io/client-go@v0.25.5 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

But k8s.io/api v0.25.5 is at go 1.19, so it normally has module graph pruning enabled — it must be getting pulled in through some other dependency.

Comment From: bcmills

Ok, here's the actual bad path: github.com/hashicorp/terraform-provider-google/v4 v4.2.0 is explicitly required and unpruned (at go 1.16).

From there, we reach the bad pprof unpruned:

github.com/hashicorp/terraform-provider-google/v4@v4.2.0 google.golang.org/api@v0.60.0
google.golang.org/api@v0.60.0 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

So it appears that in workspace mode we are failing to load the transitive dependencies of github.com/hashicorp/terraform-provider-google/v4@v4.2.0.

Comment From: bcmills

contrib/tfcore/go.mod is using a wildcard replace directive to replace google.golang.org/api with the source code of a specific version of that module:

    google.golang.org/api => google.golang.org/api v0.86.0

Comment From: bcmills

So the skew in the versions seems to come from applying the replace directives inconsistently in go work sync. - When we load the module graph for the first time, we use the replace directives from the workspace overall: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=57-60;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea - But when we later reload each module in the workspace individually, we end up using only the replace directives of the individual module: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=105-117;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea

Moreover, the error returned from EditBuildList causes go work sync to silently continue instead of erroring out: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=117-120;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea

Comment From: bcmills

So, to summarize: - There is one bug in how editRequirements applies module graph pruning when diagnosing unresolvable conflicts, which should be fixed by https://go.dev/cl/571800. - There is another bug in how go work sync applies replace directives, which will need a separate fix.

The combination of the two bugs is what produces the observed symptoms.

Comment From: matloob

We weren't able to get this done in time for 1.23, and the original error in the first issue comment isn't showing up anymore so we're going to push this to 1.24.