Go version

go version go1.22.3 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/go-cache'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/gomod-cache'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/app/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3017926117=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Set GOMODCACHE built a plugin and then tried to load it it failed with:

9 0.630 [KO] SERVER   test-plugin.so: plugin was built with a different version of package go.opentelemetry.io/otel/internal
#19 0.630 [KO] MODIFIER test-plugin.so: plugin was built with a different version of package go.opentelemetry.io/otel/internal (previous failure)
#19 0.630 [KO] CLIENT   test-plugin.so: plugin was built with a different version of package go.opentelemetry.io/otel/internal (previous failure)

If I don't set GOMODCACHE then the plugin loads as expected.

What did you see happen?

The plugin load successfully

What did you expect to see?

No errors

Comment From: seankhliao

please provide a complete reproducer

Comment From: gabyhelp

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: stevenh

This is the same as https://github.com/golang/go/issues/63230, unfortunately its closed but still not fixed. It does include a minimal reproduction case @seankhliao.

As mentioned in that issue, we don't get to control our up streams so -trimpath is not possible in our case when working with KrakenD.

As a workaround we had to refactor our docker builds to remove the use of GOMODCACHE, which isn't ideal.

A few things would be helpful until a proper fix is available: * Document -trimpath on both server and plugin as a workaround for GOMODCACHE use. * Improve the error message in this case. This was not a conflicting version, it was a issue caused by GOMODCACHE which wasn't obvious and took some hunting down.

Obviously the ideal would be the underlying issues should be fixed, but seems these issues are hard to solve as the have been around for quite some time.

Comment From: seankhliao

The plugin documentation calls this out as:

Runtime crashes are likely to occur unless all parts of the program (the application and all its plugins) are compiled using exactly the same version of the toolchain, the same build tags, and the same values of certain flags and environment variables.

It doesn't seem feasible to list out every flag that may affect loading of plugins.

Comment From: stevenh

There are flags which folks would expect to impact this GOARCH for example but I would suggest GOMODCACHE is not something anyone would think could impact the ability to load a plugin?

So unless these things are explicitly documented users will encounter unexpected issues which waste a lot of time.