Proposal Details

The x/crypto/x509roots package was added in https://github.com/golang/go/issues/43958 and https://github.com/golang/go/issues/57792 (cc @rolandshoemaker @rsc @FiloSottile who were involved in prior discussion).

One feature was discussed in issue https://github.com/golang/go/issues/43958 but did not make it into the package as currently released: accessing the x509 fallback root certificate bundle programmatically, i.e. not just using it for verification in the current process, but e.g. exporting it to a file for later usage by a different process on a different machine.

Motivation / use-case

In my case, the gokrazy packer (a Go program) creates a self-contained root file system image to be run (with the Linux kernel) on a Raspberry Pi (or similar), PC or (Cloud or on-prem) VM that only contains other Go programs. A gokrazy root file system contains no C runtime or similar — similar to FROM scratch Docker containers.

The gokrazy packer can easily be run on Linux, where we just copy the system root file into the resulting image. But on macOS and Windows, we don’t have a system root file that we can copy. That’s where we currently use github.com/breml/rootcerts.

Ideally, we would programmatically create a roots file (at “gokrazy packer time”) that the Go runtime would then load (at “Raspberry Pi run time”).

Background: Why is the x/crypto/x509roots/nss parser not sufficient?

One might wonder: Why is the x/crypto/x509roots/nss parser not sufficient for this use-case?

I originally thought that using nss.Parse might actually make implementing breml/rootcerts easier, but it turns out that breml/rootcerts already uses an approach that does not require nss.Parse: https://github.com/breml/rootcerts/blob/7000414306b0b352acb0de167dc22ebe5a584085/generate_data.go#L34

I considered doing the http.Get in the gokrazy packer, but then my program requires internet access (undesirable, especially when running in isolated CI/CD environments) and can fail when the source is slow or unavailable. So, I’ll need a cached copy and then have to deal with keeping it up-to-date.

Instead of dealing with cache management on my user’s disk, it might be better to obtain the root certs at go:generate time and embed them into my application. But then I’m effectively doing myself the work that breml/rootcerts is currently doing for me, and have not gained anything.

I think the key observation is: obtaining the root certs is not the tricky part, but updating/distributing the root certs is an annoying problem to solve. If I could just access the fallback store that the x/crypto module already contains, GitHub’s dependabot would from time to time submit a PR to update the x/crypto dependency, and that would be the easiest solution in terms of how much infrastructure I would need to maintain.

Proposal

Add the following code to x509roots/fallback/fallback.go:

// Bundle returns the fallback X.509 trusted roots as a certificate bundle.
//
// This function is primarily useful for programs that build environments in
// which Go programs should have access to the fallback roots, such as Docker
// containers.
func Bundle() []*x509.Certificate {
    return bundle
}

https://go.dev/cl/506840 is an implementation of this proposal.

Open Questions

One open question that came up during the review of https://go.dev/cl/506840:

I'm not sure "Bundle() []*x509.Certificate" is the right API for this, for example because of https://go.dev/issue/57178: in the future the bundle in fallback might include constrained roots.

We could rename to UnconstrainedRoots(). Are constrained roots going to be vital to have a functioning certificate store?

(The whole subject of constrained roots is something I’ll need to consider when working with the generator, too, which makes it less appealing to integrate at the generator level.)

Comment From: gabyhelp

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: rolandshoemaker

Ah okay, so the main use case here (at least for you) is to essentially extract the parsed NSS list but without the need to fetch it yourself, and/or manage the updating of the list, and not use it in Go itself for verification, but write it out to disk (or to an image etc). This makes sense and is mostly reasonable.

One of the reasons we didn't do this as part of the initial implementation was that it introduces some slightly complicated properties that allow for mistake making. In particular if we returned a slice of pointers, we have the problem that the user can now, inadvertently or not, mutate things within the pool itself, which I'm not sure we really want to allow/encourage. We could return values instead of pointers, but that's going to be quite expensive, since these are not small objects. Perhaps that is fine though, if this is expected to be run relatively infrequently (although that clearly constrains the value of this API).

As for constraints, we now have the constraints API (although we don't use it, and just skip anything that is currently constrained). We could just return the x509.Certificate, func(x509.Certificate) error pair, and let users decide what to do themselves, or just exclude constrained certificates from the exported bundle. I think it really depends on what we expect the use cases for this API are.

Comment From: stapelberg

Ah okay, so the main use case here (at least for you) is to essentially extract the parsed NSS list but without the need to fetch it yourself, and/or manage the updating of the list, and not use it in Go itself for verification, but write it out to disk (or to an image etc). This makes sense and is mostly reasonable.

Yes, that’s exactly right :)

As for constraints, we now have the constraints API (although we don't use it, and just skip anything that is currently constrained). We could just return the x509.Certificate, func(x509.Certificate) error pair, and let users decide what to do themselves, or just exclude constrained certificates from the exported bundle. I think it really depends on what we expect the use cases for this API are.

As discussed in person this morning, likely it would be good enough to just skip certificates that are constrained for now, as there are very few that have any constraints at all.

Also, we wondered whether the API should provide x509.Certificates at all, or just the unparsed NSS list as is currently stored in bundle.go (const pemRoots). The latter is sufficient for writing out the file to disk and more efficient to implement as well (no need to worry about modification of parsed x509.Certificates).

The new API could then be:

func PEMRoots() []byte {
    return []byte(pemRoots)
}

Comment From: rolandshoemaker

We now have implemented constraint parsing in x509roots, meaning the bundle now contains roots that should not be blindly trusted. Additionally roots are no longer stored as one giant byte slice.

I think if we are to add an API which allows accessing the bundle, it should probably also include the constraint function we generate for constrained roots. If the user doesn't want to use constrained roots they can just filter the returned certificates based on that.

I'm not particularly happy with the idea of adding this API directly to the x509roots/fallback package though, as it somewhat overloads its purpose. x509roots/fallback should exist just for registering the default pool.

That said, I think we can come up with a reasonable compromise here, which is that we add a new package, x509roots/fallback/bundle, where we put the generated bundle file and add this new API, and then use that API in x509roots/fallback to access to bundle and register the default pool.

Concretely the new API would be:

// Package bundle contains the bundle of root certificates parsed from the NSS trust store,
// using x509roots/nss.
package bundle

type Root struct {
  PEM []byte
  Constraint func([]*Certificate) error // nil if root is unconstrained
}

// Roots returns the parsed bundle of roots from the NSS trust store. Constrained roots will have a non-nil
// Root.Constraint function. All other roots are unconstrained.
func Roots() []Root

Comment From: stapelberg

Thanks, AFAICT, the suggested API would work for me :)

Comment From: aclements

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — aclements for the proposal review group

Comment From: mateusz834

With #73691 in mind, do we really want to return the PEM encoding?

type Root struct {
  PEM []byte
  Constraint func([]*Certificate) error // nil if root is unconstrained
}

I would suggest DER, so that in does not conflict with changes we want for #73691. And thus we can use the bundle package directly in the fallback package, avoiding storing both DER/PEM variants in the binary.

I would suggest:

type Root struct {
  Certificate []byte // DER-encoded certificate
  Constraint func([]*Certificate) error // nil if root is unconstrained
}

There is one question though, is the Certificate field going reference the internally stored certs (lets assume CL 676217 lands) or a clone of these bytes. The DER bundle is ~150KiB.

Comment From: rolandshoemaker

I would suggest DER, so that in does not conflict with changes we want for https://github.com/golang/go/issues/73691.

DER is reasonable, especially considering the changes in https://github.com/golang/go/issues/73691.

There is one question though, is the Certificate field going reference the internally stored certs (lets assume CL 676217 lands) or a clone of these bytes. The DER bundle is ~150KiB.

What I wouldn't give for immutable byte slices... Cloning the bytes will be quite expensive, but giving the caller the ability to accidentally (or maliciously) mutate the certificate list and cause the init of fallback to fail/accidentally load unexpected certificates would not be great (I'm not entirely sure if you could get the runtime to run the inits in the right order to do this).

Comment From: andig

What I wouldn't give for immutable byte slices... Cloning the bytes will be quite expensive

Isn't the purpose here to use these for other purposes like exporting them to disk? That will be even slower. Could an iterator be returned to pass one cert after the other instead of a big allocation?

Comment From: stapelberg

I would suggest DER, so that in does not conflict with changes we want for #73691.

DER is reasonable, especially considering the changes in #73691.

Wait, before we switch to DER, can someone confirm that Go’s crypto/x509 will load system roots in DER format transparently?

Looking at https://cs.opensource.google/go/go/+/refs/tags/go1.24.3:src/crypto/x509/root_unix.go;l=44;drc=171c794002bac46a22c74a846ef3328628ed5d49, it seems like Go expects the system roots in PEM format.

So it seems like using DER defeats the whole purpose of this proposal? (Please re-read the first post if you’re unclear.)

What I wouldn't give for immutable byte slices... Cloning the bytes will be quite expensive

Isn't the purpose here to use these for other purposes like exporting them to disk? That will be even slower.

Yes, indeed — the cost of one byte slice clone will not be measurable in the overall process of generating a dozens- to hundreds-of-MB large disk image. So, from my perspective, we can trade off security/safety over performance for this API.

Comment From: mateusz834

Wait, before we switch to DER, can someone confirm that Go’s crypto/x509 will load system roots in DER format transparently?

I would not, but you still can encode that to PEM if needed (using encoding/pem).

Comment From: mateusz834

Also one thing that might be worth considering here, is the Constraint func([]*Certificate) error, if we make this a function, then it is not serializable (if someone desired to do so). Currently AFAIK it is only a constraint of the cert dates. Maybe it should be a struct instead that has a Constraint method?

Comment From: mateusz834

And ....

If we want to be 100% safe, then we can't really use the certs from x509roots/fallback/bundle in x509roots/fallback. I was thinking that we would have an x509roots/fallback/internal/bundle which has the der encoded certs and x509roots/fallback/bundle does a clone of it, but x509roots/fallback uses it directly, but that does not work out since we expose the raw DER cert in x509.Certificate, (so the internal x509roots/fallback/internal/bundle slice can be modified, thus x509roots/fallback might misbehave, if someone modifies the der encoding through x509.Certificate). So the fallback package would need to clone these bytes, or we would have to store that bundle twice. So my argument about #73691 might no longer hold true.

Comment From: stapelberg

Hello again!

Wait, before we switch to DER, can someone confirm that Go’s crypto/x509 will load system roots in DER format transparently?

I would not, but you still can encode that to PEM if needed (using encoding/pem).

I took some time to prototype this change to ensure the proposed API works and can confirm that it does: I can indeed PEM-encode the certificates like so:

func fallbackBundle() string {
    var certs []byte
    for _, c := range bundle.Roots() {
        certs = append(certs, pem.EncodeToMemory(&pem.Block{
            Type:  "CERTIFICATE",
            Bytes: c.Certificate,
        })...)
    }
    return string(certs)
}

Also one thing that might be worth considering here, is the Constraint func([]*Certificate) error, if we make this a function, then it is not serializable (if someone desired to do so). Currently AFAIK it is only a constraint of the cert dates. Maybe it should be a struct instead that has a Constraint method?

I talked to @rolandshoemaker about this last year when we met in person, and he had some reservations regarding introducing any sort of structured Constraint API.

Using a function is the right thing to do here, I think, and the obstacle it poses for serializing the bundle.Root type is desired.

If we want to be 100% safe, then we can't really use the certs from x509roots/fallback/bundle in x509roots/fallback. I was thinking that we would have an x509roots/fallback/internal/bundle which has the der encoded certs and x509roots/fallback/bundle does a clone of it, but x509roots/fallback uses it directly, but that does not work out since we expose the raw DER cert in x509.Certificate, (so the internal x509roots/fallback/internal/bundle slice can be modified, thus x509roots/fallback might misbehave, if someone modifies the der encoding through x509.Certificate).

When you say “misbehave”, are you worried about function (HTTPS connections break) or security (malicious code injects its own certificate as untrusted)?

So the fallback package would need to clone these bytes, or we would have to store that bundle twice. So my argument about #73691 might no longer hold true.

I don’t mind the cloning cost, but it sounds like you might have stronger requirements. When you say “my argument might no longer hold true”, can you elaborate and explain which part of your comments you mean, or rephrase more clearly what you think is the best path forward please? Thank you.

Comment From: mateusz834

I don’t mind the cloning cost, but it sounds like you might have stronger requirements. When you say “my argument might no longer hold true”, can you elaborate and explain which part of your comments you mean, or rephrase more clearly what you think is the best path forward please? Thank you.

My point there is that we probably want to import (and use) the x509roots/fallback/bundle by x509roots/fallback (share these certificates in one place), and it would be nice if the x509roots/fallback use did not cause any clone, since it is executed at program startup (#73691).

So my idea was to have:

  • x509roots/fallback/internal/bundle - exposes raw DER certificates
  • x509roots/fallback/bundle - this proposal API (uses x509roots/fallback/internal/bundle and clones the bytes before returning)
  • x509roots/fallback - uses raw DER from x509roots/fallback/internal/bundle directly, without clone to save some time from program startup.

But we expose the DER field in x509.Certificate, thus someone might modify the bytes in x509roots/fallback/internal/bundle via x509.Certificate.DER, causing x509roots/fallback/bundle to return something invalid.

When you say “misbehave”, are you worried about function (HTTPS connections break) or security (malicious code injects its own certificate as untrusted)?

This could be a possible side-effect if someone modified these bytes, by misbehave i only meant that we could return in x509roots/fallback/bundle something invalid.

I am aware that this is a really extreme corner-case, as no one modifies the DER field. This is why i said:

If we want to be 100% safe (...)

Comment From: mateusz834

I think, and the obstacle it poses for serializing the bundle.Root type is desired.

I am on the side that i don't agree with that, even you noted that (and https://github.com/golang/go/issues/69898#issuecomment-2931597848):

In my case, the gokrazy packer (a Go program) creates a self-contained root file system image to be run (with the Linux kernel) on a Raspberry Pi (or similar), PC or (Cloud or on-prem) VM that only contains other Go programs. A gokrazy root file system contains no C runtime or similar — similar to FROM scratch Docker containers.

As i understand that, you want to export these certificates to the disk (image). I am aware the you are only interested in the DER of the certificates, but that shows that the use-case it to serialize that somehow and use it somewhere else (not in the Go program directly), what if at some point there is going to be some universal way to store additional constraint for certificates (alongside the PEM certificate), then you might be interested in serializing these fields to disk somehow.

But on the other hand (if that becomes needed) then we can still add more fields to Root to support that, so it might be fine going with a func for now.

Or we can make the api future-proof, in such way (Constraint does not have any fields exported):

type Root struct {
  DER []byte
  Constraint *Constraint // nil if root is unconstrained
}

type Constraint struct {/*unexported fields for now*/}

func (Constraint) Check([]*Certificate) error