Proposal Details

Background

Today, the Go standard library does not have a clear boundary between what is part of the gc-specific runtime and what is portable code. Some parts clearly are (the runtime package itself), and many parts clearly aren’t (encoding), but other packages are in a gray area.

The maphash package exemplifies this gray area. Currently, maphash has two implementations: one that is deeply tied to the Go "gc" runtime and uses the exact hash functions constructed by the compiler for use in built-in maps, and another "purego" version that reimplements hashing using reflection and a Go implementation of wyhash. We created both so that maphash would work unmodified with non-gc runtimes such as TinyGo and GopherJS.

Maphash is conceptually clearly part of the runtime, since it exports map hashing. But it is technically possible albeit awkward and slow to implement the letter of its spec in portable code.

Unfortunately, the portable implementation causes problems with the dependency graph. The non-portable version lies very low in the package dependency graph and depends on only 8 packages (go list -deps hash/maphash), all of which are extremely low level. The portable version depends on 87 packages (go list -tags purego -deps hash/maphash), mostly because it depends on crypto/rand to generate seeds and reflect to walk types.

As a result, the presence of the purego maphash means we cannot use maphash in other low-level packages such as internal/sync and unique. Even in higher level packages, this introduces tricky dependency issues.

Proposal

I propose we declare maphash to be "part of the runtime" and drop the purego implementation.

As part of this, I propose we work to be more disciplined about what parts of std are part of the runtime. For maphash, we would maintain the current separation between the pure logic in one file and the gc-specific glue code in another file. The gc-specific file would be build tagged with the “gc” build tag to clearly indicate which parts are specific to gc. This would hopefully make it easy for other Go implementations to use the package as-is and simply layer in their own file of glue code.

This would be a reversal of #47342.

Effect on other Go implementations

I believe TinyGo currently uses the purego implementation of maphash, but it has the necessary runtime functions for the gc version (rand and memhash). It uses a different type layout, which makes Comparable tricky, but does have the basic mechanism to support this.

GopherJS has its own implementation of maphash based on the Go 1.19 gc version. At the time, it needed to make significant modifications to avoid non-portable uses of unsafe. Since then, all of this code has been factored out of the main logic. It does note that this version could be dropped in favor of the purego version of maphash, but GopherJS does not require the full strictness of purego. Notably, it can and does use the GopherJS runtime random generator.

Alternatives

We could reduce the pressure on the dependency graph by rewriting the purego maphash to use math/rand/v2 (which itself is "part of the runtime" since it uses runtime.rand) and internal/reflectlite (also clearly "part of the runtime"). I believe that would reduce the number of packages to 31 (go list -deps math/rand/v2 internal/reflectlite hash/maphash). However, I believe math/rand/v2 is not currently usable in TinyGo or GopherJS, so this would still cause churn for them. I believe both implement internal/reflectlite. This approach may still limit the use of maphash in low-level packages, though to a far lesser extent, and would not address the more general problem of clearly isolating toolchain-specific code.

Comment From: gabyhelp

Related Issues

Related Code Changes

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: aclements

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — aclements for the proposal review group

Comment From: bjorndm

Apart from maphash. I think it is a great idea to give the Go standard library a clear boundary between what is part of the gc-specific runtime and what is portable code. This should make non gc-ports a lot easier to build and maintain.