Proposal Details

Go lacks a CLI to verify local repositories against the Go checksum database. Adding a flag in go mod verify to check local git tags against the sumdb would help module authors ensure their contents haven't been tampered with. This could prevent issues from unauthorized changes by someone with force-push access to GitHub, GitHub itself, or even Google.

I propose adding a -tag flag accepting the following values

all: Check all local git tags against the sumdb. latest: Check only the latest local git tag [version]: Check a specific version (e.g., go mod verify -tag=v1.0.0).

-tag=latest is especially useful as the first command to run after pushing a new tag, as it will have the side-effect of loading it in the sumdb while checking it matches the local repository.

Comment From: gabyhelp

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: ianlancetaylor

CC @matloob

Comment From: seankhliao

isn't this just GOPROXY=direct go get my.module ?

Comment From: FiloSottile

I'm pretty excited about this, so far we don't have a good story for how the author-to-sumdb link is secured, but it's a very small gap to fill thanks to the sumdb design. It also gives a canonically good answer to "how do I review a dependency knowing it's what will be used by my application" that doesn't involve vendoring or digging into the modcache.

isn't this just GOPROXY=direct go get my.module ?

Almost! That downloads the module from e.g. GitHub where it might have been tampered with. This checks the local copy as it was developed / is being reviewed. It's kind of like a hypothetical GOPROXY=local with nicer UI.

Comment From: gopherbot

Change https://go.dev/cl/596097 mentions this issue: cmd: add go mod verify -tag

Comment From: Russ741

Do we want go mod verify -tag to also verify the cache like go mod verify, or to skip that part?

Comment From: matloob

cc @ianthehat Do you have a perspective on this?

Comment From: ianthehat

I think that the functionality is a really good idea, and we should have it.

The ability to verify that the code you intended to make up a release flows through your code hosting site, into the module proxy and back down to you without modification is an important property to be able to check, and doing so is currently far too hard.

I am not totally convinced that go mod verify -tag=??? is the right interface to it, but am also happy to defer that decision to others. If it is then the help message needs more substantial rewriting than that cl (it starts with Verify checks that the dependencies of the current module, which are stored in a local downloaded source cache, have not been modified since being downloaded, and now we are making it do far more than that) We should also think if there are any other mod properties that we might want to verify in the future, and make sure the design allows for them if so.

Comment From: rsc

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

Comment From: aclements

Just for my own understanding, can someone (@ianthehat or @Esra-Al ?) explain how this would be used? It sounds like this would slot into a release process, but I'm not really sure where. Does this have to be something that's opt-in, or is there any way we can make it more automatic? What are the potential consequences of a user not doing this verification?

Comment From: ianthehat

We probably want @FiloSottile to chime in here. My guess is that the most important use is at the same point that somebody tries to tag a repository they also want to verify that the whole world sees the code they had locally when that tag is fetched. For the majority of repositories that is probably an individual developer doing it, but it could equally be a release process, or a github action, anywhere that somebody has a local copy of the repository they believe is authoritative and wants to check the full round trip. Because we don't have tooling over actually setting tags, I don't see any part of current workflows where the go command knows it should check it. We did once talk about adding functionality to help users with tagging releases (like telling them if they should do a major or minor etc), but we never moved any further with it.

Comment From: Russ741

Yeah, my understanding basically meshes with what @ianthehat posted - it can be incorporated into the release process and run to verify each new version shortly post-release.

Note that there's another use case for manual invocation here - if someone is doing a security audit of version x.y.z of a package, they can use this tool to verify that the version of the code they've checked out for inspection is the canonical x.y.z according to the go module mirror/checksum DB, rather than whatever is tagged as x.y.z in GitHub or wherever at the moment.

As far as further automation, the options I've come up with are well outside of this scope: - it might be possible to have the go module mirror periodically recheck that its cached versions of a module match what the source repo is serving up and flag mismatches (not sure what it should do about them, though) - it might be possible to set up some sort of "there's a new version of package foo.bar in the module mirror" feed for developers to subscribe to, so they find out about unexpected/malicious releases right away rather than when they initiate a valid release with the same version

Comment From: aclements

Thanks. This seems well-motivated from a security perspective, but I think it's less clear what the exact user interface to this should be. This dovetails with the question of exactly when someone/something would run this verification command. Pinging @matloob

Comment From: matloob

@ianthehat I'm interested to hear (read?) why you think go mod verify -tag may not be a good interface for this.

@Esra-Al I think if we do use go mod verify -tag as the interface, then providing the -tag flag without arguments should still do something. Could we have it do the all behavior? How much slower is all vs. latest on a repo with a lot of tags?

It seems like the alternatives to verify -tag would be (1) to implement the functionality in another tool in an x/ repo or (2) to implement it in another go mod subcommand. The main issue with (1) is that some of the vcs logic would have to be reimplemented/copied. The main issue with (2) to me is that if we do find other properties to verify, we would then end up adding new subcommands for each of them. I think there is some precedent for this type of interface with go clean which has a different behavior with and without the flags for cleaning various caches.

Comment From: aclements

Ping @ianthehat for @matloob 's question above.

Comment From: ianthehat

I am not convinced it is the right place because it has very little to do with mod files, and nothing to do with verifying a modules dependencies (which is what go mod verify does). It feels like it is just being elbowed into a fairly arbitrary spot just because that spot mentions module verification, rather than a place it actually belongs. On the other hand I don't have a good suggestion for where it really should go, or what other features might eventually join it. If we ever add features that help a user pick the right tag (we talked about this at one point, I think the suggesion at the time was go mod release), or even apply that tag, then it would kind of fit with those.

Comment From: FiloSottile

This dovetails with the question of exactly when someone/something would run this verification command.

The most common flow would be simply

$ git tag v1.2.3
$ git push --tags
$ go mod verify -tag v1.2.3

which ensures the code made it through the code hosting and Google's service, into the checksum database, unmodified.

I think if we do use go mod verify -tag as the interface, then providing the -tag flag without arguments should still do something.

If it's a string flag, how can it not take an argument?

It feels like it is just being elbowed into a fairly arbitrary spot just because that spot mentions module verification, rather than a place it actually belongs. On the other hand I don't have a good suggestion for where it really should go, or what other features might eventually join it.

I don't have a strong argument in favor of go mod verify as its place, but I think it's important to have this functionality, because at the moment we have the checksum database, clients that check for inclusion, ways to monitor unexpected entries, but then no way to actually check what's in it. That's a significant gap.

How do we move this forward?

Comment From: aclements

I agree with @ianthehat 's point that this seems almost entirely unrelated to the current behavior of go mod verify.

My sense is that this should be a new subcommand, like go mod verify-tags. I don't think it's so bad to have another subcommand. Given that we're not sure what else we would verify, I'm not worried about some future proliferation in verification subcommands. Making it a new subcommand also provides a natural way to list no tags (which could mean to verify all tags), or to list more than one tag. Depending on the behavior we want if no tags are listed, it could accept an optional -all or -latest flag, and otherwise just take a sequence of tags to check.

(Tongue-in-cheek alternative: make go mod verify take sub-sub-commands so we can say go mod verify tags or anything else we may want to verify in the future :smile:)

Comment From: ianlancetaylor

One possibility would be to add a go mod proxy command. Then we could have go mod proxy -check=TAG.

Comment From: seankhliao

what about 

go mod sum . # calculates sum for local module
go mod sum my.module@version # queries sum from sumdb

and leave the comparison up to the user?

I couldn't quite place my finger on why the proposed API felt off, but I think it's the local interaction with vcs and the name "tag" when verification should work over any version.

bonus API since go needs to do it to calculate sums:

go mod zip .

produce the module zip file, useful if you need to upload it to some more limited module host

Comment From: willfaught

If this needs to support commit hashes too, perhaps “version” should be used instead of “tag“.

Comment From: aclements

It seems like we still need to reach consensus on the actual command.

Comment From: rsc

It seems like we still need to reach consensus on the actual command.

This still seems to be the case.

Comment From: willfaught

Given the recent boltdb-go problem, it would be useful to have a way to verify that versions in the repo also match go.sum and the sum database.

Comment From: rolandshoemaker

Trying to put an end to the bike shedding, re-using go mod verify here seems like the simplest choice. The name of the command makes sense for what we're doing, and it at least somewhat lines up with what the command already does (checking integrity of things, albeit local rather than remote).

I don't expect a huge number of people are currently using go mod verify, so it's unlikely we're going to be confusing a lot of people, and we can update the documentation to make it clear this command exists for verifying various properties of a local module (i.e. integrity of it's local dependencies, that the local version matches the proxy/upstream version, perhaps other things in the future).

Comment From: aclements

Based on the discussion above, this proposal seems like a likely accept. — aclements for the proposal review group

The proposal is to extend the go mod verify command with a new -tag=x flag that performs tag verification by checking local git tag x against the sumdb. This helps module authors ensure their contents haven't been tampered with between their local machine and the sumdb. Two special values of x would be supported: -tag=all would check all local git tags against the sumdb, and -tag=latest would check only the latest local git tag.

This would typically be used as part of the release process, when pushing a tag. For example:

git tag v1.2.3
git push --tags
# Verify that our local v1.2.3 tag matches sumdb.
go mod verify -tag v1.2.3

Comment From: thepudds

As proposed, is it correct that this would not directly help with #66653?

(I lost the thread of this discussion a bit... I looked over it again to try to catch back up, but wanted to explicitly confirm the relationship to #66653).

I don't expect a huge number of people are currently using go mod verify, so it's unlikely we're going to be confusing a lot of people, and we can update the documentation to make it clear this command exists for verifying various properties of a local module (i.e. integrity of it's local dependencies, that the local version matches the proxy/upstream version, perhaps other things in the future).

If go mod verify was used to help improve #66653 at some later date, is the thinking that it might just be one of the "various properties of a module" that might be added to also check for the current module when someone does go mod verify (which might be expensive), or maybe it might mean there could be new flag to go mod verify that says to also check the remote repos for all dependencies in the build graph (or some other definition of "all"), or something like that?

(We don't need to solve all problems at the same time, and I salute @rolandshoemaker's effort to move things forward here, but just checking to confirm there are at least plausible extensions to what is now "likely accept" here).

Comment From: seankhliao

I think it helps for #66653 if you downloaded the source repo pointed to by pkgsite and ran go mod verify -tag to check that it matched what pkgsite / proxy saw.

it doesn't help if you stay in the browser, I don't think any go cli can help with that.

Comment From: thepudds

I think it helps for https://github.com/golang/go/issues/66653 if you downloaded the source repo pointed to by pkgsite and ran go mod verify -tag to check that it matched what pkgsite / proxy saw.

Hi @seankhliao, if I'm a module author and I want to check the dependencies of my module for an issue like https://github.com/golang/go/issues/66653, I think that would be downloading the repo for each dependency of my module (for example via git clone) and then running go mod verify -tag <proper-tag> for each of those repos, or something like that?

So that helps, and if go mod verify -tag moves forward as proposed, someone might hopefully publish a small community tool to help automate those steps, and/or people will do various bash one-liners or similar...

But I guess I'm hoping for a single command as part of cmd/go (maybe go mod verify <something> or go <something-else>) under the theory that the more convenient it is, the more people will do it...

(As I said, though, I did lose the thread of this discussion, so maybe that doesn't make sense, or maybe it is already better covered than I currently understand).

Comment From: seankhliao

If you wanted to verify building from the source of all your dependencies, then it should be just: GOMODCACHE=$(mktemp -d /tmp/goproxy.XXXX) GOPROXY=direct go mod download: * clean module cache to not use a locally cached version * direct to not use a proxy cached version * verifying the checksum against sumdb is automatic when retrieving dependencies

I think go mod verify would be unsuited for managing direct downloads of dependencies.

Comment From: aclements

As proposed, is it correct that this would not directly help with https://github.com/golang/go/issues/66653?

As others have said, this isn't trying to fix #66653 and that's okay. 😊

Comment From: aclements

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — aclements for the proposal review group

The proposal is to extend the go mod verify command with a new -tag=x flag that performs tag verification by checking local git tag x against the sumdb. This helps module authors ensure their contents haven't been tampered with between their local machine and the sumdb. Two special values of x would be supported: -tag=all would check all local git tags against the sumdb, and -tag=latest would check only the latest local git tag.

This would typically be used as part of the release process, when pushing a tag. For example:

git tag v1.2.3
git push --tags
# Verify that our local v1.2.3 tag matches sumdb.
go mod verify -tag v1.2.3

Comment From: Russ741

What's the next step in the process? @matloob mentioned that https://go-review.googlesource.com/c/go/+/596097 could be reviewed once the proposal was accepted.

Comment From: aclements

I think https://go-review.googlesource.com/c/go/+/596097 just needs to be reviewed and landed.

Comment From: samthanawalla

@Esra-Al Thank you for your CL! If you have time, could you also send another CL to update x/website via https://go.googlesource.com/website

Comment From: dylan-bourque

Given the recent boltdb-go problem, it would be useful to have a way to verify that versions in the repo also match go.sum and the sum database.

Reading the latest comments here and it seems like this might not actually provide this functionality. I don't see any mention of go mod verify -t going all the way to VCS to re-verify that the value in the sumdb actually matches the content at that tag.

The BoltDB issue was that the malicious code was pulled into the proxy and sumdb (and subsequently cached) then the tag was moved in the repo. Consumers were seeing the malicious code as valid and verified because both the proxy and sumdb agreed with the local cache on the contents of that module, but nothing was going all the way back to GitHub.

I don't see any test that exercises the case where the tag is moved after the code has been pulled into the proxy and sumdb. Am I missing something?

Comment From: seankhliao

As I understand, checking your remote vcs is out of scope. Instead you'd check out a copy of the from the remote locally and verify that (it also means what you use / look at is what you check), otherwise the remote end could serve different content.

Checking all your dependencies is also out of scope for this command, for that, use GOMODCACHE=$(mktemp -d /tmp/goproxy.XXXX) GOPROXY=direct go mod download.