Golang proposal: cmd/go: allow go:cgo_import_static outside of the standard library

Proposal Details

The go:cgo_import_static directive is used to import unresolved references from an object file that has been statically linked with the Go binary. It is used in the standard library, for example, to load symbols from the race detector C library, which is also statically linked.

Note that, contrary to what the name suggests, this directive doesn't need cgo, the Go internal linker is smart enough to statically link most of the C code out there.

It is currently only permitted in the standard library and in cgo-generate code, although it was allowed everywhere until it was restricted as part of the #23672 fix. The reason to restrict this particular directive is not well documented, as it can't be used to execute arbitrary code when running the Go toolchain.

I need it because I'm maintain some thin C bindings over system Swift functions, and I want to allow the users of these bindings to use it without requiring cgo nor installing any separate library. The only way to achieve this is compiling the bindings as a static object file, and then load all the necessary objects from it using go:go_import_static.

@golang/compiler @golang/security @golang/proposal-review

Comment From: ianlancetaylor

I'm a little puzzled because you mention the Go internal linker, but go:cgo_import_static is documented as only affecting external linking. What is the failure mode when you don't use the directive?

Comment From: qmuntal

cgo_import_static does work with the internal linker, the documentation is a bit outdated. The only barrier to using it is the Go toolchain rejecting this directive outside of the standard library and cgo files. This is the error I get when using it:

go:cgo_import_static only allowed in cgo-generated code

And this is what I want to achieve, being foo a symbol provided in an static object file:

//go:cgo_import_static foo
var foo byte

//go:linkname foo foo
var foo byte

func Foo() {
  syscall.Syscall(uintptr(unsafe.Pointer(&go_hashSize)))
}

Comment From: cherrymui

If I understand correctly, this would allow user code to access C symbols (in the precompiled C object) without cgo. But on most platforms, we don't have a general mechanism to call a C function from Go without using cgo. E.g. marshaling arguments in C ABI, and doing stack switches and alignments, etc.. If we do this, it probably would encourage user to write hacky unsafe code for that. That doesn't seem like a good idea.

I think there is a general demand for calling C functions in precompiled (static or dynamic) objects, which I think is definitely worth considering. But I think it needs a more complete solution, e.g. a general mechanism to call a C function. Also, on some platforms, it may require the program to be initialized in certain way, e.g. using pthread to create threads, instead of direct syscalls.

Comment From: qmuntal

If I understand correctly, this would allow user code to access C symbols (in the precompiled C object) without cgo.

Yes, this is my main motivation to use cgo_import_static.

But on most platforms, we don't have a general mechanism to call a C function from Go without using cgo.

That's a good point. I'm mostly interested in supporting this for windows and darwin, which do have good support for C calls without cgo.

If we do this, it probably would encourage user to write hacky unsafe code for that. That doesn't seem like a good idea.

True, people will use it in unsafe code. My only counter argument here is that people are already doing similar unsafe things with go:cgo_import_dynamic, which is allowed outside of the standard library, so allowing this new directive won't put us on a much worse situation.

But I think it needs a more complete solution, e.g. a general mechanism to call a C function.

Can't wait for it to happen 😄.

Comment From: ianlancetaylor

What happens when using the internal linker if you omit the //go:cgo_import_static directive? I'm wondering why the //go:linkname directive is not enough.

Comment From: qmuntal

What happens when using the internal linker if you omit the //go:cgo_import_static directive? I'm wondering why the //go:linkname directive is not enough.

Touché. Just tested removing the directive and nothing broke. Now I'm confused 😓, why the go:cgo_import_static directive exists then?

I'm going to retract this proposal then given @cherrymui feedback and @ianlancetaylor suggestions. I'm leaving it open for a day in case someone wants to add more into the discussion.

Comment From: ianlancetaylor

When using the external linker, the Go linker generates an object file that is passed to the external linker. The //go:cgo_import_static directive tells the Go linker that it should emit the named symbol as an undefined symbol in the object file, rather than giving an error that the symbol is undefined.

Comment From: ianlancetaylor

By the way, the names cgo_import_static and cgo_import_dynamic are terrible. They refer to an early iteration of cgo that required using shared libraries for the C code when using what we now call external linking.