What version of Go are you using (go version)?
$ go version cgo version go1.19.11
Does this issue reproduce with the latest release?
This issue does reproduce with the latest release.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/dbernadett/.cache/go-build" GOENV="/home/dbernadett/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/dbernadett/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/dbernadett/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/home/dbernadett/.cache/bazel/_bazel_dbernadett/f2fcdd1f2e8b2102ba2af9a91a4f07d8/external/go_sdk" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/home/dbernadett/.cache/bazel/_bazel_dbernadett/f2fcdd1f2e8b2102ba2af9a91a4f07d8/external/go_sdk/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.19.11" GCCGO="gccgo" GOAMD64="v1" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/dbernadett/Nuro/go.mod" GOWORK="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2772875216=/tmp/go-build -gno-record-gcc-switches"
What did you do?
I ran cgo with a large number of arguments which exceeded my computers argument limit for fork/exec. This is a common problem and usually addressed by allowing arguments to be specified through a params file.
What did you expect to see?
I expect to see a flag, such as -param which allows arguments to be passed to the cgo executable. This would be very similar to how it is implemented here: https://github.com/bazelbuild/rules_go/blob/4211c6d32ee475a8fde1cfc91571e7c0bed67af4/go/tools/builders/compilepkg.go#L45
What did you see instead?
Instead, there is no params flag and calling the cgo executable with too many params on some platfoms causes fork/exec to fail.
Comment From: seankhliao
can you give an example invocation?
Comment From: dbernadett
echo "go run main.go -ldflags=\\" > command.sh
for i in {1..10000}; do echo "'-lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread'\\" >> command.sh ; done
echo "'-lpthread'" >> command.sh
bash command.sh
The above will work for a trivial hello_world go program if the number of -lpthread args is small. However if we overrun the limits on the mac getconf ARG_MAX as found in this stack overflow post this is the output:
dbernadett@dbernadett-laptop seperate % go version
go version go1.21.3 darwin/arm64
dbernadett@dbernadett-laptop seperate % bash gen_command.sh
command.sh: line 1: /usr/local/go/bin/go: Argument list too long
Comment From: cherrymui
By "the cgo executable" you mean the cmd/cgo command? But the example you gave in https://github.com/golang/go/issues/63882#issuecomment-1789818731 is passing very long ldflags to the go command, not the cgo command. Why do you need to pass very long flags to cgo? Can you use environment variable GOFLAGS, CGO_CFLAGS, CGO_LDFLAGS, or #cgo directives in the source file instead?
Comment From: dbernadett
Here would be a real example with cgo which also fails:
echo "go tool cgo -- \\" > command.sh
for i in {1..10000}; do echo "-lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread \\" >> command.sh ; done
echo "main.go" >> command.sh
bash command.sh
Here is an example of it failing with env_vars on mac
echo "go tool cgo -- \\" > command.sh
echo "main.go" >> command.sh
CGO_LDFLAGS=""
for i in {1..10000}; do
CGO_LDFLAGS+="-lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread "
done
export CGO_LDFLAGS
bash command.sh
The cgo directive is neat, but seems impractical for use by the rules_go bazel ruleset.
Comment From: cherrymui
Thanks. It is unclear why you need a large number of flags, though.
Also, the flags are mostly to pass to the C toolchain, could you just write a response file and pass @response and the cgo tool will pass it through?
Comment From: dbernadett
I am working for a company which is using bazel to build our go programs. It is more valuable to us (and probably other companies) that rules_go can just pass a mass of arguments to the cgo command instead of spending a bunch of time figuring out what flags are neccessary, and then how to ensure only the necessary flags are emitted by the go_library bazel rule.
What is a response file? Could you provide either an example or some link to documentation.
Comment From: cherrymui
https://gcc.gnu.org/wiki/Response_Files
https://man7.org/linux/man-pages/man1/gcc.1.html search for @file on the page
Comment From: dbernadett
echo "go tool cgo -debug-gcc -- \\" > command.sh
echo "main.go" >> command.sh
CGO_CFLAGS="@arg_file.txt "
for i in {1..2}; do
CGO_CFLAGS+="-lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread "
done
export CGO_CFLAGS
bash command.sh
bash gen_command_env_vars.sh > debug.txt 2>&1
When I run the above commands, my debug.txt doesn't seem to have any references to -lpthread. Would you know how to modify this example to get them to show up in the gcc calls?
Comment From: dbernadett
Just an update, working on verifying that @param file works for me.
Comment From: cherrymui
go tool cgo -debug-gcc doesn't seem to show the flags, regardless of whether @param is used or not. go tool cgo -debug-gcc doesn't compile the C code anyway, so I'm not sure whether/why you want to pass those flags. Try go build -x.
Still, could you show a real example that you need to pass a very long command line (not a synthetic one that pass -lpthread many times)?
Comment From: dbernadett
Unfortunately I am working on proprietary code, but suffice it to say that we are using the go compiler through bazel. Is there a reason you need a concrete example?
Comment From: dbernadett
I could probably get a sample invocation, but it woulnd't be the easiest thing to do.
Comment From: dbernadett
Here is another description of the bug from the rules_go folks https://github.com/bazelbuild/rules_go/issues/2654
Comment From: dbernadett
Here is the line that generates a "way too long" CGO_LDFLAGS environment variable. https://github.com/bazelbuild/rules_go/blob/f2d409bc52d4945f283e89f8b501cbab65c83fbb/go/tools/builders/cgo2.go#L142 Maybe at this point we could create a params file and send it to cgo using the @paramfile syntax?
Comment From: gopherbot
Timed out in state WaitingForInfo. Closing.
(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)
Comment From: ianlancetaylor
Whatever problem remains here is probably fixed in the 1.23 release by https://go.dev/cl/584655.