Go version

go version go1.22.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE='auto'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/romain.marcadier/Library/Caches/go-build'
GOENV='/Users/romain.marcadier/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/romain.marcadier/go/pkg/mod'
GONOPROXY='github.com/DataDog'
GONOSUMDB='github.com/DataDog,go.ddbuild.io'
GOOS='darwin'
GOPATH='/Users/romain.marcadier/go'
GOPRIVATE='github.com/DataDog'
GOPROXY='binaries.ddbuild.io,proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD=''
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/qh/q2mpbd0j6xsb7vc8dqzdm81h0000gn/T/go-build2570734933=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Trying to use go tool covdata textfmt -i <some-dir> -o <some-report.txt> results in errors such as:

error: error: opening counter data file coverage/raw/covcounters.848fe7b4490389df1276a71be50dd0d9.79237.1721148957056415000: open coverage/raw/covcounters.848fe7b4490389df1276a71be50dd0d9.79237.1721148957056415000: too many open files

This is the case when trying to consolidate coverage reporting from thousands of individual reports, which my use-case generates.

What did you see happen?

The errors above, which happen on all go releases in our test suite so far: - go1.21 - go1.22 - go1.23 release candidate

What did you expect to see?

I expect the tool to produce a report without needing to retain all files open at once.

Without having validated this claim (yet), I am pretty sure the culprit is this code:

https://github.com/golang/go/blob/8b48290895d362417124bcc18f0e6b6476ddc99e/src/cmd/internal/cov/readcovdata.go#L208-L239

Where the files are open in a loop that includes a defer statement to close the file; but the defer only happens at function exit, and the loop might go over a large number of files.

I would suggest refactoring the loop body to a new function/method as a way to reduce the lifetime of the open file descriptor here.

I might be able to submit a change request to gerrit (but not today).

Comment From: gabyhelp

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: cherrymui

cc @thanm

Comment From: gopherbot

Change https://go.dev/cl/598735 mentions this issue: cmd/internal/cov: close counter data files eagerly

Comment From: thanm

Thanks for the report.

I've send a CL that should fix this issue. If you can test it yourself with Go tip that would be helpful, thanks. If there are issues, please go ahead and re-open the issue.

Comment From: RomainMuller

Hey, can confirm gotip now works without bailing out on too many files open!

Will this CL be back-ported on 1.21 and 1.22 as well? (I'd like this, but I can deal with it if not).

Comment From: thanm

I can request backports, but it is up to the release team to approve. Generally the rule is that we backport only for serious bugs (compiler crash, compiler generates incorrect code, runtime crash, etc) with no workaround, but we'll see.

Comment From: thanm

@gopherbot please consider this for backport to 1.22 and 1.21.

Comment From: gopherbot

Backport issue(s) opened: #68491 (for 1.21), #68492 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

Comment From: gopherbot

Change https://go.dev/cl/599036 mentions this issue: cmd/internal/cov: close counter data files eagerly

Comment From: gopherbot

Change https://go.dev/cl/599055 mentions this issue: cmd/internal/cov: close counter data files eagerly