Golang go/src/cmd/go/internal/vcs || src/cmd/go/internal/list : VCS errors with ssh blocked on private repositories

Go version

1.24

Output of `go env` in your module/workspace:

`go env` is not relevant

What did you do?

Attempted to use gorelease, which fails during the following command: go list -x -m -versions -- mygitserver.net/${PROJECT PATH}

In the real data, the url is deemed a "schemeless" repo, since its not in the list of known hosts.

In this instance, the CI Runner only supports HTTPS for temporary tokens and does not have single instance ssh keys available.

What did you see happen?

https://github.com/golang/go/blob/8552bcf7c261cd150d0074c4ec7e2412b20af0a5/src/cmd/go/internal/vcs/vcs.go#L1223

Sanitized CLI output from CI runner:

$ go list -x -m -versions -- mygitserver.net/my-sub-folder/ci_sampleapp
# get https://mygitserver.net/my-sub-folder/ci_sampleapp?go-get=1
# get https://mygitserver.net/my-sub-folder/ci_sampleapp?go-get=1: 200 OK (0.108s)
# get https://mygitserver.net/my-sub-folder?go-get=1
# get https://mygitserver.net/my-sub-folder?go-get=1: 200 OK (0.068s)
mkdir -p /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs # git3 ssh://git@mygitserver.net:2222/my-sub-folder.git
# lock /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f.lock
mkdir -p /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f # git3 ssh://git@mygitserver.net:2222/my-sub-folder.git
cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git init --bare
0.004s # cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git init --bare
cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git remote add origin -- ssh://git@mygitserver.net:2222/my-sub-folder.git
0.003s # cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git remote add origin -- ssh://git@mygitserver.net:2222/my-sub-folder.git
cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git ls-remote -q origin
0.122s # cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git ls-remote -q origin
go: module mygitserver.net/my-sub-folder/ci_sampleapp: git ls-remote -q origin in /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f: exit status 128:
    Host key verification failed.
    fatal: Could not read from remote repository.

    Please make sure you have the correct access rights
    and the repository exists.

What did you expect to see?

It should determine if SSH or HTTP is required based on previous operations in the go list command; then use the scheme determined to perform the git remote add origin command.

Comment From: seankhliao

I believe this is misconfiguration on your side somewhere, because it works by default.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

Comment From: UnaffiliatedCode

Hi Sean, I do believe this quantifies as a bug. If you click the link provided above with the exact line of code creating the bug, the author explicitly called out the assumption that git (default SSH) is always available (since git is the first scheme in the list of possible vcs communication schemes). SOURCE

// If we know how to ping URL schemes for this VCS,
// check that this repo works.
// Otherwise, default to the first scheme
// that meets the requested security level.

The ping command is : "ls-remote {scheme}://{repo}"

This command is not outputted to console.

The easiest way to replicate locally is to use GIT_ALLOW_PROTOCOL locally

GIT_ALLOW_PROTOCOL="https"

This will cause the a FATAL to occur; if you were able to replicate my CI runners environment.

Due to the nature of some self-hosted git instances, they do not all use SSH keys; some use short lived Tokens for operations. In some cases, a git config of the following is used to explicitly force https over ssh. This is not necessary for this bug, however it is a means of replicating the issue locally on your environment.

NOTE: git config is done in a non-GLOBAL capacity intentionally.

git config url."https://USER:${TOKEN}@mygitserver.net/my-sub-folder".insteadOf "git@mygitserver.net/my-sub-folder"

In the linked code-base, you can see references to standardized bypasses for this behaviour here: Link: https://github.com/golang/go/blob/8552bcf7c261cd150d0074c4ec7e2412b20af0a5/src/cmd/go/internal/vcs/vcs.go#L1559

These explicit bypasses are used to enforce https as the default for known hosts; but private instances are unknown hosts.

Further more, if you look at the above output from the ci runner; earlier operations in the go list operation, you can see it uses https for authentication, successfully.

 get https://mygitserver.net/my-sub-folder/ci_sampleapp?go-get=1
# get https://mygitserver.net/my-sub-folder/ci_sampleapp?go-get=1: 200 OK (0.108s)
# get https://mygitserver.net/my-sub-folder?go-get=1
# get https://mygitserver.net/my-sub-folder?go-get=1: 200 OK (0.068s)

However, later, it uses ssh to initialize a repository locally (despite SSH never being used previously).

cd /builds/my-sub-folder/ci_sampleapp/.go/pkg/mod/cache/vcs/25abf4eed0e2466cf46798d3b42c9640a5d3e282ac04af8e502298937df00d7f; git remote add origin -- ssh://git@mygitserver.net:2222/my-sub-folder.git

The specific line linked, succeeds on the ping (due to https being used as opposed to ssh); then instantiates a git init in a new repository within a temporary folder.

This git init command loses the configuration from the previous git repository; and initializes an SSH based communication; which fails.

Additionally, I can fix this issue via a local fork by doing either of the following (which are not sustainable solutions based on the other assumptions within the codebase):

Re-order the default protocols to have https first, as opposed to git || ssh
Apply the assumption that the go list command is being run within a git repository
this means we can use the same git config that already exists either in the new repo, or use the repo in place to perform the operations.