Golang x/pkgsite: API for pkg.go.dev

Prior to pkg.go.dev, godoc.org has had a JSON API that can be used to, among other things, discover importers of a given package.

Example: https://api.godoc.org/importers/golang.org/x/net/html

Given that pkg.go.dev does a much better job at tracking importers thanks to Go Modules and the Module Proxy, it would be nice if the community could get access to a public API similar to that of godoc.org.

Comment From: julieqiu

@rhcarvalho - it would be helpful to get a sense of what your current use cases are for api.godoc.org, and feature requests are for an API for pkg.go.dev.

It sounds like getting the importers for a package is one of them. With pkg.go.dev being module aware, what specific information about importers would be useful to surface via an API?

For example: - importers for a specific version of a package - only importers of the latest version of a package - any importer for all versions of a package - something else?

Additionally, what other information would be useful to you to surface via an API?

/cc @tbpg who has also mentioned wanting an API for pkg.go.dev

Comment From: rhcarvalho

Without much thought, having an API to answer the more specific question "what are the importers of a specific version of a package" would make plausible to derive the answer to the other items in your list.

At the moment I consume the godoc.org API and scrape data from pkg.go.dev to answer the question "who uses my package". As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Here are the other endpoints in the GoDoc API: https://github.com/golang/gddo/blob/7365cb292b8bfd13dfe514a18b207b9cc70e6ceb/gddo-server/main.go#L901-L904

• /search: takes a q parameter. Can be useful to explore known packages/modules.
• /packages: attempts to return all known packages, no pagination. Doesn't seem too usable as is.
• /importers/: useful, hard to compute without central knowledge.
• /imports/: dispensable, easy to compute offline with go list.

So if we need a more specific request, here it is:

api.go.dev MVP

• /importers/{module-or-package}/@v/<version>: returns the list of importers of a given module/package at a given version. URL scheme could be tuned to match the Go Proxy specification (go help goproxy).
• /search/: matches the functionality of https://pkg.go.dev/search?q=hello, returns JSON instead.

Comment From: tbpg

Another endpoint idea (standalone or as part of a set of info returned for a module): available versions of a given module.

Comment From: myitcv

@tbpg

Another endpoint idea (standalone or as part of a set of info returned for a module): available versions of a given module.

Shouldn't people use cmd/go for that? Not least because of the fall-through semantics of the go env GOPROXY` variable. Noting https://github.com/golang/go/issues/37367

Comment From: julieqiu

What information beyond https://proxy.golang.org/<module>/@v/list would be provided by that endpoint?

Comment From: tbpg

Shouldn't people use cmd/go for that? Not least because of the fall-through semantics of the go env GOPROXY` variable. Noting #37367

There are some cases where cmd/go might not be available and a normal web API would be helpful. I think it's reasonable to assume pkg.go.dev will only return the modules/versions it knows about.

What information beyond https://proxy.golang.org/<module>/@v/list would be provided by that endpoint?

None. That works. :)

Comment From: myitcv

There are some cases where cmd/go might not be available and a normal web API would be helpful

I'd be wary of encouraging tool authors to query pkg.go.dev/a proxy directly. Because it could well introduce skew compared to the answers from cmd/go.

Comment From: adamdecaf

In my use-case I don't want to execute cmd/go binaries as I just want to grab some metadata (i.e. detected license of all dependencies) from a dozen modules and organize it on a webpage. If I need local clones of those modules it really complicates this internal tool.

Comment From: myitcv

In https://github.com/golang/go/issues/37952 I raised the question of whether module/package license file information could be surfaced in the output of cmd/go list. Following a conversation on yesterday's golang-tools call, we concluded that doing so would be a bad idea; reasons summarised in https://github.com/golang/go/issues/37952#issuecomment-611527845.

@ianthehat instead suggested exposing the information via a pkg.go.dev API, leveraging the fact that the content and presentation on pkg.go.dev has already jumped through the relevant legal hoops.

This comment is therefore to explicitly request that we include license file information in the API. Thank you!

Comment From: tiegz

👋 Thanks for all the new Go tools like pkg.go.dev, they're super useful. Some input on this topic from the perspective of Libraries.io:

• +1 to a /search/ endpoint, or maybe just adding pagination to this url so it's possible to get a list of all packages: https://index.golang.org/index?since=TIME&limit=LIMIT. Many popular package repositories offer a full listing like this, e.g. https://packagist.org/packages/list.json for Packagist

• What information beyond https://proxy.golang.org//@v/list would be provided by that endpoint?

Currently to get the publish times for all versions, you have to make N+1 requests: one for @v/list and then @v/xxx.info for each version and grab the "Time" value for each one. Assuming it's difficult to change the Go proxy spec, an API endpoint that returns all versions with metadata would be really nice.

• As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Anyone know if there are plans to fix this?

Comment From: jba

• As far as I can tell the data in the "Imported By" go.dev tab is unrelated to the version of the package I'm currently browsing.

Anyone know if there are plans to fix this?

No immediate plans. We currently gather that information from import statements in the code, so there is no version information attached. The go.mod file doesn't have all the version information we need.

So we understand it's an approximation and we want to fix it, but it's going to take some time.

Comment From: pombredanne

@julieqiu my use for an API would be to: 1. search and list packages 2. for a package, access all the information as available for a given package in each of the tabs, such as here https://pkg.go.dev/github.com/gin-gonic/gin?tab=licenses ... basically everything that is available as HTML in the frontend https://github.com/golang/pkgsite/tree/master/internal/frontend should be available in a JSON api

Comment From: pombredanne

@julieqiu oh and please do not retire http://api.godoc.org/packages unless there is an alternative!

Comment From: eclipseo

it would be helpful to get a sense of what your current use cases are for api.godoc.org, and feature requests are for an API for pkg.go.dev.

As a downstream package maintainer for Fedora, I'm also interested in an API. We have our own tool, Anitya, to track package releases, but it was not designed to track GIT commits. And many Go packages still don't publish version. So any information about latest published commit, with info like date of the commit, new dependencies, would be very helpful. I would gather data from the API in Python and compare it to the latest version we have for a given package.

I'm also interested in getting the License info, so we could find recursively all the licenses used in a static binary.

Comment From: pombredanne

and BTW my general use case is for https://github.com/nexB/scancode-toolkit and related projects to provide my users with license, origin and dependencies details. And for https://github.com/nexb/vulnerablecode where we provide vulnerabilities details.

Comment From: StevenACoffman

My present use case is discovering major version upgrades. Currently, we have to scrape pkg.go.dev. An alternative to pkg.go.dev API, would be if there were an "official" way (via go list for example) in the future to retrieve the latest major version of a module, we could use it in tools like go-mod-upgrade and icholy/gomajor but that would depend on golang/go#40357 (deprecated notification) and golang/go#40323 (notify about newer major versions). Thanks!

Comment From: icholy

@StevenACoffman I've updated gomajor to find newer versions using the module proxy. The performance is acceptable.

Comment From: 0x2b3bfa0

My use case is importing modules/packages to another package manager (Guix) where builds are completely reproducible and monolithic vendoring is strongly discouraged. Having a JSON API would simplify the obtention of metadata like licenses, brief package descriptions, dependencies and hashes.

Comment From: jba

We'll be keeping api.godoc.org around for a while.

Comment From: jba

Our current plan is to keep api.godoc.org around until we have a suitable replacement.

Comment From: myitcv

/search/: matches the functionality of https://pkg.go.dev/search?q=hello, returns JSON instead.

Building on this point, it would be great if a real-time fuzzy search API were exposed. That way command line tools could wrap/use this.

Comment From: restuwahyu13

Hello everyone, is there a public API for pkg.go.dev ?, if there is I want to use it for my package library i.e. gopack-cli, for now I'm using the go doc API, for downloading go package library from gopack-cli .

Comment From: jba

@restuwahyu13 , keep using api.godoc.org.

Comment From: restuwahyu13

But is api.godoc.org the same as pkg.go.dev sir @jba ?

Comment From: fzipp

@restuwahyu13 pkg.go.dev doesn't have an API yet, that's why this issue is open and not closed.

Comment From: shellscape

Stopping by to ask that any first version of an API provide some means to glean the source code / repo location of a package that uses a vanity path/url/name.

Comment From: guseggert

I have similar use cases as others, so I resorted to scraping the UI as well (:nauseated_face:), and put it behind an API/CLI: https://github.com/guseggert/pkggodev-client

Comment From: aj-arena

The /packages api endpoint, https://api.godoc.org/packages, is returning {"error":{"message":"Internal Server Error"}} .

Has this endpoint been migrated?

Comment From: tills13

Seems like most api.godoc.org endpoints are erroring, currently?

e.g. https://api.godoc.org/packages, https://api.godoc.org/imports/github.com/goburrow/cache, etc.

I'm particularly interested in the imports endpoint. Has this moved somewhere else or is there a mirror I can use?

Comment From: pombredanne

This is starting to be really problematic. @jba could someone kindly look into this? This is really looking bad

Comment From: pombredanne

@julieqiu do you know what happened?

Comment From: jba

Will get on this today.

Comment From: pombredanne

@jba you rock... that's very kind of you! :bow:

Comment From: pombredanne

Note that only alternative today would be to resort to extensive pseudo-random screen scraping of https://pkg.go.dev/search laced in with some extensive recursive hitting of https://index.golang.org/index to get a complete package picture. It can work if this is what is recommended, but it would feel a bit like it is 1999 all over again. ;) If anything and if that's the way to go, then at least I could create a common service that does this for everyone?

Comment From: jba

api.godoc.org should be back up.

Comment From: pombredanne

@jba this is responding but not doing anything for me:

$ wget https://api.godoc.org/packages
--2022-02-01 14:33:56--  https://api.godoc.org/packages
Resolving api.godoc.org (api.godoc.org)... 142.250.179.147, 2a00:1450:400e:80f::2013
Connecting to api.godoc.org (api.godoc.org)|142.250.179.147|:443... connected.
HTTP request sent, awaiting response...

this stays there hanging waiting, no I/O nor CPU used.

Comment From: pombredanne

@jba strike this out. It completed after a while! Thank you ++ :bow:

Comment From: ZiViZiViZ

Prior to pkg.go.dev, godoc.org has had a JSON API that can be used to, among other things, discover importers of a given package.

Sorry for the stupid question but where are the documentations for using this "JSON API" that godoc.org had?

Comment From: srenatus

@ZiViZiViZ https://github.com/golang/gddo/wiki/API

Comment From: ZiViZiViZ

Thanks but this does not seem to have enough information. I am looking to be able to get the license for a package. I was hoping there is a way to do something like that:

 $ curl -s https://api.godoc.org/search?q=gopkg.in/yaml.v2@v2.4.0 |jq '.results[].license'
"Apache License 2.0"

Comment From: tills13

@ZiViZiViZ the way I accomplish this is by making a request to https://pkg.go.dev/gopkg.in/yaml.v2?tab=licenses and using an HTML parser to grab the license headers. (replace gopkg.in/yaml.v2 with whichever package you're looking for the license for)

Comment From: ZiViZiViZ

Thank you. That is what I am doing right now but I thought there might be a better way to do this.

Comment From: Alphare

I am part of the Software Heritage initiative to archive all versions of public code and packages.

I'm looking for a way of listing all go packages, planning then to download the source for each release. For example in the Rust ecosystem, crates.io has an API that allows us to get all the metadata for a known package and each of its releases, which is very practical.

It appears that there only exists https://api.godoc.org/packages which returns a single 300MB+ JSON file containing only URLs to the repositories of packages (and some import_count metric that is always 0), but nothing more. https://index.golang.org/index is not looking any different. Am I missing anything worthwhile that would help?

Thanks in advance.

Comment From: seankhliao

please keep this issue on topic and refer questions to https://github.com/golang/go/wiki/Questions

Comment From: shellscape

Rather than hiding what I'd consider a lot of really useful comments by folks (because I landed here for many of the same reasons they did, and there is extremely little available with regard to go package APIs) it would be infinitely more useful to explain to folks how to use the wiki for questions - or specify that tangential and related questions here are not welcome, and folks should seek the answers on the wiki. As it is now, that curt reply seems to imply that we should post questions on the wiki itself, which I doubt is what you're actually after.

The go team has been kicking this can for years, so users are naturally going to have a lot of questions. This reply https://github.com/golang/go/issues/36785#issuecomment-579820081 specifically asks users what features they're after in an API - thus the issue is fair game for feature-related questions. Go packages are the very last major ecosystem package registry to lack a comprehensive API and that which doesn't make its data readily available. Because of this, people are naturally going to have a myriad of questions about trying to access that data. I'd humbly ask @seankhliao to be a bit more gracious in your moderation of this issue.

Comment From: golightlyb

• Use case: generating static documentation, or running a local documentation server, but wanting to be able to query (with javascript, at the time of the page view by a web browser) the "[Imported by: NNN]" number that appears if instead viewed on pkg.go.dev.

• currently api.godoc.org/importers is returning {"error":{"message":"Internal Server Error"}}

Comment From: rsc

For what it's worth, we have been running api.godoc.org as an unmonitored service (meaning we don't page people for it), approximately "best effort" although even that may be too generous. We discovered today that it has been down for 30+ days because the disk filled. Given that being down for a month was a non-issue, we've decided to leave it down.

The godoc.org redirects will continue to run (approximately forever), but api.godoc.org will no longer be accessible. Or rather it will continue to be inaccessible.

Comment From: golightlyb

@rsc fair enough if this is the decision but the previous comment is mine, right at the start of that 30+ day period, saying it's down, and three people agreeing.

If this is the decision, again, fine - but I submit that it shouldn't be based on people not complaining because to be fair I raised this pretty much as soon as it happened and it was never fixed and didn't have a response (fair enough- priorities) so anyone saying the same thing wouldn't be adding anything, and nobody knew they had a 30ish day deadline to object

(I don't intend for this to sound harsh, just as a disagreement! 🙂)

Edit: as an aside that might be helpful for people with an interest in this, Google's experimental Open Source Insights has an API. I wasn't aware of it until recently but it could very well resolve this entire conversation.

Comment From: shellscape

honestly this reflects pretty poorly on the entire team and the project in general. compared to other ecosystems and their package management, and user experience, the go community leaves a lot to be desired.

Comment From: shellscape

Nice. Comment on the actions taken, or lack thereof, and get hidden as off-topic. If the team isn't open to polite and honest criticism, lock the topic.

Comment From: gwatts

I use the Dash doc viewer on my Mac which i find super useful for building a local index of all sorts of docs, Go included - Just found i couldn't install a new Go docset into it, presumably as it can no longer build an index from api.godoc.org.

Would be nice to point the author towards an alternate index if this one isn't returning as the usefulness of Dash is greatly diminished for me without having rapid access to the docs of Go packages i use frequently.

Comment From: AlekSi

Dash 6.3.1 now relies on the web scraping. It works, but API would be better.

Comment From: hyangah

FYI deps.dev launched a new API service.

https://docs.deps.dev/api/v3alpha/

Its GetPackage, GetVersion, GetDependencies may be useful for users who need the list of versions, license info, and dependency info. I think it also provides vulnerability info as well. (cc @adg)

This does not cover the /search, /packages, /importers endpoints of the old gddo though. (OTOH, it's unclear to me what people want to see from /packages endpoint given the current volume of data)

Comment From: avamsi

I was looking for an API to search for Go "commands" -- https://deps.dev//search/suggest?system=go&kind=package&q=axl (and then https://api.deps.dev/v3alpha/systems/go/packages/github.com%2Favamsi%2Faxl) kinda works (I don't mind that deps.dev//search is hacky and potentially unstable), but pkg.go.dev also annotates the commands as such (see https://pkg.go.dev/search?q=axl, for example), so it's easy to tell them apart from vanilla libraries.

Comment From: sfblackl-intel

https://docs.deps.dev/api/v3alpha/

We are using that...but realized that for Go packages in particular, there is no publish date...which led me to this thread. My main real ask is that publish date be included for Go packages...otherwise my solution appears to be to call something like this for each package/version (https://proxy.golang.org/<module>/@v/v1.9.0.info).

Comment From: dvob

I also used api.godoc.org for my project https://github.com/dvob/go-project-usage/ To obtain the importers I now scrape https://pkg.go.dev/%s?tab=importedby (see https://github.com/dvob/go-project-usage/blob/72c5625c361b7e81fa2d1a35591c30830dc1164f/main.go#L134). Unfortunately this is prone to break (e.g if CSS class name changes) and for packages with many importers it only returns 20k packages.

Comment From: xnuinside

I want to woke up this tread. https://docs.deps.dev/ that is referencing before does not contain any information about standard libraries packages (https://pkg.go.dev/std). Answering on why and who will use API: all researchers & data miners that needed to analyze open source packages (huge set of tools for code analyzes, code efficiency & vulnerabilities) (and it is only from my work-domain, I believe that there are more purposes) - we need information about package when we resolve vulnerabilities, when we investigate how languages growth, that dynamic on them, we need this information to track new releases & fixes & etc. Mostly all popular ecosystems for languages (maven, pypi, crates, conan, conda, etc) & operation systems like linux provide API to get this information - it could be DB dump, it could be REST API, github repo, anything, it could be huge csv files - any format will be better when scraping website.

UPD: also, there is no ANY informations about packages (subpackages?) inside repo, for example, https://pkg.go.dev/vuln/GO-2021-0073 for Affected Package - github.com/git-lfs/git-lfs/lfsapi on deps.dev info only about 'root' packages like github.com/git-lfs/git-lfs and you cannot find any information that packages inside

Comment From: mark-pictor-csec

Another use for the API would be to check if any versions of a module are retracted. Right now the only way I see to get that info is pretty terrible - I must GET pkgs.go.dev/path/to/pkg/?tab=versions, and search the html for the word retracted. If it's there then I get to parse the html and figure out which version(s) are so marked.

People at my company are very thankful that go vulnerabilities usually include affected symbols (example), as it allows us to report to customers whether they're actually using a vulnerable function with far better accuracy.

So while vuln reports are much better in go, for retractions we must parse a web page and hope the html doesn't change. This feels like a real step back.

I hope an API can be prioritized.

Comment From: nabokihms

+1 for the feature, I'd like to use pkg.go.dev to find the repo with the source code by the package name. Now I must parse the HTML code to do so.

Comment From: seankhliao

I think getting vcs info is #18387

go list -m -f '{{.Origin}}'

Comment From: seankhliao

re: retractions, parsing go list -json -retracted or the go.mod directly with https://pkg.go.dev/golang.org/x/mod@v0.22.0/modfile#File seems to make much more sense than going through pkgsite, it'll even work for private dependencies.

Comment From: xxxserxxx

My use case for this is services like shields.io. pkg.go.dev already parses out and processes a lot of Go-specific information that could be used to generate end-user badges in READMEs. It also produces a single piece of information that is -- AFAIK -- not available elsewhere: the number of packages that import a given package. Additional useful information already being parsed:

• The package version
• The software license
• The number of imports

Without running bespoke services to re-do this data extraction that pkg.go.dev is already doing, the alternative for most users is to parse this information out and then commit this information to some repository; this has long been a source of contention for VCS users -- for example, if committed to the project repo itself, it muddies the VCS history.

pkg.go.dev is already extracting all of this information -- making the metadata that it composes available via an API call (or calls) would allow use in badge creation and potentially other useful analysis tools.

For my money, all I'd ask for is a single API that provides the metadata processed and compiled by pkg.go.dev in a single JSON or even TOML response. shields.io already supports generating shields from API calls that return a number of different formats.

Comment From: josharian

Is the status on this issue that it needs a policy decision on whether any API-ish access is allowed, or a case by case discussion?

I re-found this this morning, hoping for JSON output from /search.

If it’s in case-by-case mode, I can file a sub-issue. If it’s a policy question, I guess I just join the waiting queue. :)

Comment From: jba

It's a policy/bandwidth/priority thing.

Comment From: josharian

Would CLs be reviewed?

Comment From: jba

Not at this time, I'm sorry.

Comment From: anistark

Is there any update on apis for https://pkg.go.dev/ ?

Comment From: jba

@anistark There continue to be no plans for an API.

Comment From: acheong08

I'm curious how the Repository is detected in the details. I don't see any references to it in repositories or go.mod. e.g. gopkg.in/natefinch/lumberjack.v2 resolves to github.com/natefinch/lumberjack. There is a mention in the readme but I've also seen some without mentions still getting tagged with the current repository. If it's some internal information derived from private information, I'd like to see an API for that.

Comment From: xplshn

I would suggest a way to identify if a package is a program or a library, it would be really helpful