Golang cmd/go: test cache uses stale coverage data with -coverpkg

Go version

go1.26-devel_bb124921e9 Sun Jul 27 12:36:07 2025 -0400 darwin/amd64

Output of `go env` in your module/workspace:

╰─ go env
AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/ryancurrah/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/ryancurrah/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/fk/8lgm5_252mb0ln68l_z9wlz80000gn/T/go-build3317734112=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/Users/ryancurrah/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ryancurrah/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/ryancurrah/git/github.com/ryancurrah/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/ryancurrah/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/ryancurrah/git/github.com/ryancurrah/go/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.26-devel_bb124921e9 Sun Jul 27 12:36:07 2025 -0400'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Hello guys,

Thanks a lot for resolving https://github.com/golang/go/issues/23565.

I started testing this feature and encountered a problem that occurs when I use the go test command with the -coverpkg flag.

When coverage is collected with the -coverpkg flag and test result caching is enabled, the generated report may include lines that no longer exist.

This happens because: - Each test package attempts to collect coverage for all packages matching the -coverpkg pattern. - If the test result is loaded from the cache, the coverage data may be outdated—especially if the test package does not directly or indirectly depend on the modified code, leaving the cache uninvalidated.

For example we have project with layout:

Project layout

proj/
  some_func.go
  some_func_test.go
  sub/
    sub.go
    sub_test.go
  sum/
    sum.go

Files Content

some_func.go

package proj

import "proj/sum"

func SomeFunc(a, b int) int {
    if a == 0 && b == 0 {
        return 0
    }
    return sum.Sum(a, b)
}

sub.go

package sub

func Sub(a, b int) int {
    if a == 0 && b == 0 {
        return 0
    }
    return a - b
}

sum.go

package sum

func Sum(a, b int) int {
    if a == 0 {
        return b
    }
    return a + b
}

some_func_test.go

package proj

import (
    "github.com/stretchr/testify/require"
    "testing"
)

func Test_SomeFunc(t *testing.T) {
    t.Run("test1", func(t *testing.T) {
        require.Equal(t, 2, SomeFunc(1, 1))
    })
}

sub_test.go

package sub

import (
    "github.com/stretchr/testify/require"
    "testing"
)

func Test_Sub(t *testing.T) {
    t.Run("test_sub1", func(t *testing.T) {
        require.Equal(t, 0, Sub(1, 1))
    })
}

Coverage result of this tests

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok      proj    (cached)    coverage: 44.4% of statements in proj/...
ok      proj/sub    (cached)    coverage: 22.2% of statements in proj/...
    proj/sum        coverage: 0.0% of statements

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.22 1 1
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

Change sub.go a bit

sub.go

package sub

func Sub(a, b int) int {
    if a == 0 && b == 0 || a == -100 {
        return 0
    }
    return a - b
}

Coverage result after change

go test -coverpkg=proj/... -coverprofile=cover.out ./proj/...
ok      proj    (cached)    coverage: 44.4% of statements in proj/...
ok      proj/sub    0.005s  coverage: 22.2% of statements in proj/...
    proj/sum        coverage: 0.0% of statements

mode: set
proj/some_func.go:5.29,6.22 1 1
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 1
proj/sub/sub.go:3.24,4.22 1 0  <- Old (Should have been invalidated and removed)
proj/sub/sub.go:4.22,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 1
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 1
proj/some_func.go:5.29,6.22 1 0
proj/some_func.go:6.22,8.3 1 0
proj/some_func.go:9.2,9.22 1 0
proj/sub/sub.go:3.24,4.35 1 1  <- New
proj/sub/sub.go:4.35,6.3 1 0
proj/sub/sub.go:7.2,7.14 1 1
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0
proj/sum/sum.go:3.24,4.12 1 0
proj/sum/sum.go:4.12,6.3 1 0
proj/sum/sum.go:7.2,7.14 1 0

What did you see happen?

The report merges cached and fresh coverage, so you end up with two sub.go:3.24 entries, one that is stale and one that is correct. That duplication trips up tools that emit Cobertura XML.

What did you expect to see?

I expected the report to keep only the latest coverage and invalidate the cached coverage reports.

Comment From: ryancurrah

I have a change with a test that reproduces the issue with a fix https://github.com/golang/go/pull/74773.

Comment From: gopherbot

Change https://go.dev/cl/690775 mentions this issue: cmd/go: invalidate test cache when -coverpkg sources change

Comment From: hihoak

@ryancurrah Hi, as I understand it, these changes would disable caching for projects that use the -coverpkg flag with patterns like -coverpkg=github.com/my/project/... (assuming this pattern includes all files in the repository)

Caching would only work if no files were modified

If this is the intended behavior, it could become problematic. In Go 1.24, the default coverage collection excludes lines that were hit by tests from imported packages, which often forces to specify -coverpkg (e.g., -coverpkg=github.com/my/project/...). As a result, these changes would significantly reduce the benefits of caching for many projects https://github.com/golang/go/issues/23565

Comment From: hihoak

@ryancurrah Maybe we should leave changes from issue https://github.com/golang/go/issues/23565 as they are?

However, there's another issue we need to address if we keep these changes - specifically regarding how go tool cover -func and go tool cover -html work.

Currently, if a file doesn't exist in the coverage report, the command fails with this error:
cover: open /builds/my/project/internal/some.go: no such file or directory

Here are the relevant source code links for this behavior (-func and -html flags):
- https://github.com/golang/go/blob/master/src/cmd/cover/func.go#L78
- https://github.com/golang/go/blob/master/src/cmd/cover/html.go#L48

To fix this problem i prepared small changes https://github.com/golang/go/pull/74929, I can create an issue to fix this behavior.

Comment From: ryancurrah

That seems like a bandaid fix. We should really invalidate the invalid coverage references.

Comment From: ryancurrah

I will take a look and see if its possible to only invalidate coverage report related cache.

Comment From: hihoak

@ryancurrah I’ve thought about it, and it seems to me that with the current implementation of the -coverpkg flag, it’s impossible to fix this bug in better way than you did. Because anyway currently -coverpkg includе all files of each package matching the pattern in the coverage report for every test—regardless of whether the test depends on them or not.

It appears this issue with stale coverage data could be solved another way. In fact, all these ideas attempt to restore the old behavior of how Go collected coverage before Go 1.24. To remind you, how it worked by default: tests collected coverage not only from the current test package but also from every package imported by the code under test.

I have a couple of ideas:

1) Change the behavior of the -coverpkg flag - what if we modify how this flag works? Currently, it collects and includes all packages matching the pattern in every test’s coverage report. Instead, we could redesign it to follow this logic: it would only collect and include packages that are directly or indirectly imported by the test being run and match the pattern provided in -coverpkg. This optimization would also speed up the go test command for large projects, as it would avoid unnecessary coverage analysis for unrelated packages

2) Restore the old coverage collection behavior through another flag - this would likely conflict with -coverpkg, meaning the two flags couldn’t be used together.

Comment From: ryancurrah

@hihoak have a peek at https://github.com/golang/go/pull/74773. I updated the PR description. I think this might be a good solution.

Comment From: hihoak

@ryancurrah Hi, maybe I misunderstood, but the issue I highlighted in this comment is still open: https://github.com/golang/go/issues/74873#issuecomment-3164364047

The problem is that caching efficiency for repositories using the -coverpkg flag would be dramatically reduced.

Comment From: ryancurrah

The improvement in the change is that only coverage report cache is now invalidated not the test cache. Also just now I improved it some more by only hashing the cover packages once during each test execution.

Comment From: hihoak

@ryancurrah Sorry, maybe I don’t understand, but will it force a rerun of all tests anyway?

Comment From: zgzhong

We implemented feature #23565 in our company before it was officially resolved. When we ported the test coverage functionality to Go 1.22, we encountered the same issue. Previously, the result cache with -coverprofile worked fine before Go 1.22.

After some investigation, I discovered that the issue stems from the coverageredesign feature introduced in this change list: https://go-review.googlesource.com/c/go/+/495452.

Prior to coverageredesign, when running tests with the -coverpkg flag, all matching packages were linked together into the test binary. This meant that changes to sub.go were handled correctly.

However, with the new coverage computation, the process now relies on actual code coverage and the CoverMetaFile (for packages that are not direct dependencies). Specifically, the coverage rate for the sum package is calculated as $LineHitOf\_Sum/(LineInstrumentOf\_Sum+LineMetaOf\_Sub)$

Currently, the caching mechanism uses the OutputID (a hash of the test binary or build artifacts) as the key to retrieve cached results. As a result, when the sub package changes, the sum package still receives the old coverage result from the cache, which contains the old implement of sub

To address this issue, I think that the CoverMetaFile should be considered when computing the cache key.

Comment From: ryancurrah

@hihoak this change will not force all tests to rerun no. In fact I'll add a test case to ensure that is true.

Comment From: ryancurrah

@zgzhong interesting, I will have a look at CoverMetaFile.

Comment From: ryancurrah

I havent forgot about this, plan on working on it this weekend and investigating @zgzhong feedback.

Comment From: ryancurrah

Also note we are starting to see this issue pop up in our environment as we started allowing teams to use Go 1.25 today.

failed to merge coverage files [/workspace/repo/coverage.unit.out /workspace/repo/coverage.stack.out]: failed to add profile: OVERLAP MERGE: go.acme.dev/widgetmaker/internal/healthcheck_processor.go

We forked gocovmerge and modified it to remove the duplicate coverage reports as a mitigation for now.

Comment From: ryancurrah

Our fix worked but now golang.org/x/tools/cover.ParseProfiles does not work. I'm going to take a different route and use a custom build of golang with my patch.

Comment From: ryancurrah

Ok we deployed the patch in https://github.com/golang/go/pull/74773 to our build environment and the coverage issues are now resolved, ~but coverage caching is not working. Will look into it more tomorrow.~

EDIT: After more testing, locally and in ci, the cache is working as expected with the patch and without cover profile enabled. We had to clear our ci caches for it to start working again, or wait for the old cache to expire.

Comment From: ryancurrah

@matloob we have been running this change in production for 2 weeks now and it appears to solve our issues. Can you review https://go-review.googlesource.com/c/go/+/690775?

Comment From: JanneAalto

Hi, just checking in to see if there are any updates on this issue. We're experiencing the same problems.

Comment From: ryancurrah

Still trying to get someone to look at my pull request. I'll try on the Gophers Slack on Monday.

Golang cmd/go: test cache uses stale coverage data with -coverpkg

Go version

Output of go env in your module/workspace:

What did you do?

Project layout

Files Content

some_func.go

sub.go

sum.go

some_func_test.go

sub_test.go

Coverage result of this tests

Change sub.go a bit

sub.go

Coverage result after change

What did you see happen?

What did you expect to see?

Output of `go env` in your module/workspace: