Proposal Details

cgo proposal to avoid "argument list too long" (E2BIG) issues when CGO_LDFLAGS outgrows system limits.

Problem Description

The cgo command writes the contents of the CGO_LDFLAGS environment variable into the archive. C/C++ linker flags and dependencies are passed via this environment variable.

When there is a large list of C/C++ dependencies, it is easy to hit the system size limits, causing compilation to fail with cgo: argument list too long. - On Linux, the maximum size of an environment variable string is limited by MAX_ARG_STRLEN (128kiB). - Windows has a similar limit.

Related issues

  • https://github.com/bazelbuild/rules_go/issues/2654
  • https://github.com/golang/go/issues/63882
  • https://github.com/golang/go/issues/42295

Overly large lists dependencies can be written into a response file, using the @file syntax, passing CGO_LDFLAGS=@<path-to-file> instead of failing the compilation.

This is a convention supported by many compilers, such as gcc or clang, and mirrors the passLongArgsInResponseFiles fix for long argument lists in #18468. Multiple go commands also support response file arguments via objabi.Flagparse.

Proposal: Since build systems such as bazel may execute compile and link stages in different locations, cmd/cgo should - expand the contents of an @response_file, - the existing expandArgs function can be used to do this.

Comment From: rsc

What is the specific change? Cgo already understands response files for command line arguments; the problem here is that the info is being passed to cgo in $CGO_LDFLAGS, and that's exacerbated by the go command and Bazel rules_go both not deduplicating the CGO_LDFLAGS info they gather from every cgo-using package in the build. (If 10 packages all say the same CGO_LDFLAGS, no need to say that part 10 times, just say it once.) Perhaps they should both deduplicate and then the length limit is not a problem anymore.

We may need to deduplicate some linker arguments to appease the Mac linker anyway.

Comment From: grrtrr

Our CGO_LDFLAGS got huge, using a response file avoided the overflow. Can post the PR we used, this fixed the problem.

I had tried de-duplication, but there were some ordering dependencies in the linker command.

Comment From: rsc

How did you use a response file at all? Response files are for command-line arguments, not environment variables.

Comment From: rsc

If we need to do something, we should add a -ldflags argument to the cgo command, and then that can be put in a response file. Interpreting @ signs in environment variables is a big scary door to open.

Comment From: rsc

The linked bazelbuild/rules_go#2654 has an actual CGO_LDFLAGS line in the top comment. It is one line but it is very very long and GitHub has added a scroll bar, so you don't notice how long it is.

That CGO_LDFLAGS has:

  • 843 .lo files, each wrapped in -Wl,--whole-archive ... -Wl,--no-whole-archive.
  • 207 .a files, without that wrapping.
  • 75 -pthread
  • 4 -lpthread
  • 3 -lm
  • 2 -ldl
  • 1 -static-libstdc++
  • 1 -static-libgcc
  • 1 -lrt
  • 1 -l:libstdc++.a
  • 1 -fuse-ld=/usr/bin/ld.gold
  • 1 -Wl,-z,relro,-z,now
  • 1 -Wl,-no-as-needed
  • 1 -B/usr/bin

So deduplicating is not going to cut it. My suggestion is we add -ldflags to cgo, and then the go command and Bazel can use the support cgo already has for response files on its command line.

Comment From: grrtrr

How did you use a response file at all? Response files are for command-line arguments, not environment variables.

In short, CGO_LDFLAGS accepts @file in the same way as an executable would , expanding the arguments from the given file.

Support for this in cgo is needed to fix the corresponding open problem in @rules_go (https://github.com/bazelbuild/rules_go/issues/2654): - @rules_go collects all the linker flags into CGO_LDFLAGS and then - calls cgo with this environment in go/tools/builders/cgo2.go.

The fixed environment-variable length limit on Linux/Windows makes it currently impossible to compile larger projects.

Thinking about whether -ldflags would help - there is also a maximum commandline length.

Comment From: rsc

Thinking about whether -ldflags would help - there is also a maximum commandline length.

It would help because cgo accepts response files on the command line already.

Comment From: grrtrr

Sorry for taking so long to understand - if adding -ldflags is an equivalent solution to passing via CGO_LDFLAGS in @rules_go, then this is clearly preferable.

Comment From: rsc

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

Comment From: grrtrr

Thank you.

Comment From: rsc

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long.

Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem.

Comment From: rsc

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group

The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long.

Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem.

Comment From: gopherbot

Change https://go.dev/cl/584655 mentions this issue: cmd/cgo, cmd/go: add cgo -ldflags option, use it in cmd/go

Comment From: fmeum

The PR that closed this issue ended up breaking rules_go (and other external users of cgo) since CGO_LDFLAGS is now silently ignored in 1.23, even if the new -ldflags is not set.

Could this be fixed in a patch release? If that isn't possible, what would you recommend as a workaround?

Cc @ianlancetaylor

Comment From: grrtrr

@fmeum - the problem is in rules_go, #2654, quoting @jayconrod:

this can't be fixed in rules_go without a corresponding fix in the Go toolchain. Consider filing a bug in golang/go for that.

The "fix in the Go toolchain" was this proposal and the PR that folllowed.

The next step is updating @rules_go to take advantage of the -ldflags.

This should not be difficult to do, happy to collaborate.

Comment From: fmeum

The next step is updating @rules_go to take advantage of the -ldflags.

This part of the PR looks good to me, thanks! We can adopt it.

The problem is that existing versions of rules_go no longer work with Go 1.23 due to cgo no longer honoring the env variable. See https://github.com/bazelbuild/rules_go/issues/3979 for the reported breakage this causes.

Comment From: ianlancetaylor

I will send a patch to cgo to use both the CGO_LDFLAGS environment variable and the -ldflags command line option.

We should presumably be able to remove that when we no longer need to support older versions of Bazel.

Comment From: gopherbot

Change https://go.dev/cl/596615 mentions this issue: cmd/cgo: read CGO_LDFLAGS environment variable

Comment From: ianlancetaylor

Thanks, this should be fixed in the next release candidate. If you are able to, please test Go repo HEAD.

Comment From: fmeum

@ianlancetaylor Thanks for the quick fix!

@grrtrr Happy to review a change to rules_go to make use of the new flag.