Golang proposal: reflect: package reflection

Proposal

I'd like to propose basic package reflection in reflect to support discovery of types, functions, and variables in a package.

Similar to how value reflection allows discovery of methods and fields given a struct value; package reflection should allow discovery of functions, variables, and types given a package.

Problem

This closes a gap that exists today where code using reflect must be "pre-loaded" with the types it should know about, making every new type, function, or variable that "wants" to be reflected first register itself with the reflector. This is not up to the standard of how reflection of structs works, where reflecting code can iterate over every struct field and method.

Example Usage

One possibility is to retrieve a package from an "anchor" type:

import "reflect"
import "time"

t := reflect.TypeOf(time.Time{})
p := t.Package() // Returns a reflect.Package

Another, which uses a modified syntax, allows direct retrieval:

import "reflect"
import "time"

p = reflect.PackageOf(time) // Returns a reflect.Package

Example Interface

A reflect.Package might respond to similar calls as a reflect.Value:

type Package interface {
    // Equal reports true if p is equal to q.
    // For two invalid packages, Equal will report true.
    // Otherwise, Equal will report true if p and q
    // refer to the same package.
    Equal(q Package) bool
    // Function returns a function value corresponding to p's i'th function.
    // Function panics if i is out of range.
    Function(i int) Value
    // FunctionByName returns a function value corresponding to the function
    // of p with the given name.
    // It returns the zero Value if no function was found.
    FunctionByName(name string) Value
    // NumFunction returns the number of functions in the package p.
    NumFunction() int
    // NumType returns the number of types in the package p.
    NumType() int
    // NumVar returns the number of fields in the package p.
    NumVar() int
    // PkgPath returns a defined package's path, that is, the import path
    // that uniquely identifies the package, such as "encoding/base64".
    Path() string
    // String returns a string representation of the package.
    // The string representation may use shortened package names
    // (e.g., base64 instead of "encoding/base64") and is not
    // guaranteed to be unique among types.
    // To test for package identity, compare the Packages directly.
    String() string
    // Type returns a type corresponding to p's i'th type.
    // Type panics if i is out of range.
    Type(i int) Type
    // TypeByName returns a type value corresponding to the type
    // of p with the given name.
    // It returns the zero value if no type was found.
    TypeByName(name string) Type
    // Var returns the i'th variable of the package p.
    // It panics if i is out of range.
    Var(i int) Value
    // VarByName returns the package variable with the given name.
    // It returns the zero Value if no variable was found.
    VarByName(name string) Value
}

Comment From: ianlancetaylor

This makes it impossible for the linker to discard unused functions and variables, which it does today. I don't think this is realistically feasible.

Comment From: mvdan

To further drive @ianlancetaylor's point: this would make most Go binaries significantly larger, considerably hurting https://github.com/golang/go/issues/6853.

Comment From: glacials

Is this different than what happens for unused struct methods and fields?

Comment From: thediveo

The "unused struct fields" aren't removed because they are typically needed in order to interface with syscalls, C libraries, et cetera. Such fields might be used for padding or could be left-overs from syscall API changes over time. In any case, I don't think that you can compare unused struct fields with unused functions. If your proposal would be implemented as is, then my Wireshark plugin would be multiple times its current size, because all of a sudden there would be "tons" of Azure and Kubernetes API client code included, even if not used at all.

Maybe you might would like to be able to discover only the functions and types that actually end up in a binary, because they are used? Maybe more akin to this thwd's answer to How to discover all package types at runtime?

Comment From: glacials

I see, thanks for explaining. For the problem I'm trying to solve, which is something akin to first- and second-party plugin management, the packages being excluded from the binary would be a show-stopper.

So it sounds like this is a nonstarter, although I'm curious to hear if there are practical alternatives outside of having the importing package maintain a list of explicit references to names it needs, even when all further usage is through metaprogramming.

Options I've looked at are [plugin](https://pkg.go.dev/plugin) which creates too much fragility for us, and [github.com/hashicorp/go-plugin`](https://github.com/hashicorp/go-plugin) which seems complex for our needs.

Comment From: Splizard

This makes it impossible for the linker to discard unused functions and variables, which it does today. I don't think this is realistically feasible.

Only when the package is used for reflection, if the package is not used in this way, unused functions and variables can be discarded as usual. Similar to runtime reflection data, which isn't stored if the value is never stored within an interface value.

Comment From: ianlancetaylor

If one small package somewhere uses this new facility, then the linker can't discard anything from any package anywhere. That's a hazard for any large Go project.

Comment From: Splizard

If one small package somewhere uses this new facility, then the linker can't discard anything from any package anywhere. That's a hazard for any large Go project.

If you can call Package() on arbitrary reflect.Type values, sure. Easy adjustment to the proposal is to disallow this and restrict package reflection to explicit package selection calls ie. reflect.PackageOf(time).

Comment From: earthboundkid

Re: plug-ins, this talk was interesting and lists several ways of doing it: https://youtube.com/watch?v=pRT36VqpljA

Comment From: Jorropo

Currently trying to pass a package identifier as anything gives use of package time not in selector compile-time error. PackageOf would be an adhoc rule of this which I would find confusing.

As far as I can think about only unsafe and builtin are allowed that luxury of special language rules right now, both since they happen to be created before the existence of generics and unsafe because having it return constants is just too useful and you should know what you are doing anyway if using it. IMO in this case the gains does not outweigh the cost of an other std exception to the language rules.

Comment From: ianlancetaylor

I see that you suggested reflect.Package(time) in the original post, but I don't understand why a program would want to get a list of functions in a known package. What is the use case for that?

Comment From: earthboundkid

ISTM, it would probably be better as reflect.Package("time") because apart from the technical issues with bare package names, then you can do pkgName := discoverPlugins(); pkg := reflect.Package(pkgName).

Comment From: Jorropo

@carlmjohnson This would be opening the door to package names being passed at runtime. Preventing almost all code pruning. What if someone pass a package that is not used at all by the binary, should the runtime download sources from proxy.golang.org, build it in memory, and load the result JIT like ?

You could make the signature always require a const string but that is not a thing the language offers and would be yet an other exception.

Comment From: thediveo

@Jorropo please don't overshoot here, this comes across as trying a little bit too hard to push the original request over the cliff. Instead, let's try to see the chances here.

From my own experience with plugin management of statically build-in plugins I can imagine a benefit of being able to discover the actually linked-in modules, and then their exported types (again, only the actually included ones).

However, I'm unclear as to how to ensure that the plugin functions to be exported actually get linked in, as just underscore importing their containing packages isn't sufficient? Does this mean that this idea of package inspection would fall flat because the desired pruning would cause the p,ugin functions to not even being linked into the final binary?

Comment From: sblackstone

There are some use cases for being able to get the name of the package the code originated from - logging.

I often use a function like pkgLogger() which just returns a zerolog instance with the package name set... This could be cleaned up significantly if I didn't need to reimplement it all over the place.

Comment From: earthboundkid

runtime.Frame.Function already contains the package name of a stack frame that you can use for logging. I guess the scenario you're talking about is logging based on the package of a struct instead of the package of a function?

Comment From: kcross-ctoken

Hmmm ... I think this would be better done by adding some built in package functions to be able to iterate over public members of the package.

import "reflect"
import "time"

//IT would be nice if we can use generics but 🤷 
type Variable[T] struct{
  String Name
  Reflect.Type
  ... some hidden stuff to be able to get and set.
}
func ( v * Variable[T]) Get() iter.Seq[*T] {}
func ( v * Variable[T]) Set( value T) {}

for var := range time.Variables() {
  ....
}

// Similar with constants and functions 
// time.Functions()
// time.Constants()

Comment From: apparentlymart

Reflecting on this quite some time later (since the previous comment put it back on my radar 😀 ) it seems like the motivating use-case is not very well defined and so it's hard to know what alternative solutions might be acceptable.

There is some high-level idea about it being for runtime-loaded plugins discussed in https://github.com/golang/go/issues/61796#issuecomment-1668667236, but with plugin systems I've encountered before there's typically some expectation of each plugin implementing some specific API rather than just trying to enumerate all of the symbols declared in it.

I'd be interested to learn more about the details of the use-case to understand to what extent it's necessary for the unit of analysis to be an entire package rather than, for example, a struct type containing a mix of normal data fields and function pointer fields, or some other thing that package reflect already knows how to analyze.

I'm asking this only in the hope that it might move the discussion forward to concrete alternatives that have better feasibility than whole-package analysis.