The x dependencies vendored into the standard library (via src/go.mod and src/cmd/go.mod) should be updated for each code freeze, so that we can apply any needed fixes to those dependencies without also pulling in unnecessary changes or requiring significant backporting work (see, for example, #36851).
This task, like #11811, #12042, and the API audit (#36167, #32813), should occur regularly in each development cycle. Probably we should update the dependencies at the beginning of each code freeze, and again when we reopen the tree at the end of each code freeze.
CC @golang/osp-team
Comment From: gopherbot
Change https://golang.org/cl/217517 mentions this issue: all: update module dependencies
Comment From: dmitshur
This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping.
Comment From: bcmills
Completed the early-in-cycle update for the 1.15 cycle. Removing the label so we remember to revisit at the close of the cycle.
Comment From: gopherbot
Change https://golang.org/cl/231657 mentions this issue: all: update module dependencies
Comment From: dmitshur
Completed the start-of-freeze update for the 1.15 cycle in CL 231657. Moving to 1.16 milestone and adding early-in-cycle.
Comment From: gopherbot
Change https://golang.org/cl/234002 mentions this issue: cmd: update golang.org/x/mod to v0.3.0 (same commit)
Comment From: dmitshur
This issue is currently labeled as early-in-cycle for Go 1.16. That time is now, so this is a friendly ping so the issue is looked at again.
Comment From: dmitshur
I'm creating a short program that performs this task. Using a program makes it easier to precisely define our update policy, and adjust it over time as needed. It also makes this task less error prone and easier to review. Finally, this can help with #41409 and #36852.
The policy I'm implementing right now is informed by the strategy we used in past manual CLs (e.g., CL 231657), with small adjustments. The policy is for all modules in the GOROOT/src directory (currently, std and cmd), update all direct (not indirect) modules with the import path prefix "golang.org/x/" to their latest tip pseudo-version (which can later become tagged as a release version).
The small adjustment is that the github.com/ianlancetaylor/demangle module won't be automatically updated as part of this process. My rationale is that it's different enough that it probably warrants being updated on its own schedule, when there is an explicit need. /cc @ianlancetaylor
Indirect golang.org/x dependencies are not updated, although they could be. I'm going with the more conservative choice for now. I'm open to feedback on this, and can see pros/cons to both strategies.
I've also considered but left out other modules in the GOROOT tree (e.g., GOROOT/misc) for now.
Comment From: dmitshur
Indirect golang.org/x dependencies are not updated, although they could be. I'm going with the more conservative choice for now. I'm open to feedback on this, and can see pros/cons to both strategies.
Looks like we did update indirect dependencies last time. If we don't, it leaves much fewer dependencies to update, because the go.mod files of golang.org/x modules aren't being updated very often.
Will think more about it and likely go with what we did last time as the next step here.
Comment From: gopherbot
Change https://golang.org/cl/255859 mentions this issue: misc/update: add program to update golang.org/x dependencies
Comment From: gopherbot
Change https://golang.org/cl/255860 mentions this issue: all: update vendored dependencies during Go 1.15 development
Comment From: gopherbot
Change https://golang.org/cl/256357 mentions this issue: cmd/updatestd: add program to maintain standard library modules
Comment From: gopherbot
Change https://golang.org/cl/256538 mentions this issue: cmd: update github.com/ianlancetaylor/demangle
Comment From: dmitshur
CL 256357 has updated golang.org/x/... packages, and CL 248499 (/cc @hyangah) will update the remaining packages (pprof and demangle).
We'll need to repeat this at start of code freeze for Go 1.16, so removing early-in-cycle label.
Comment From: gopherbot
Change https://golang.org/cl/248499 mentions this issue: cmd/vendor,cmd/pprof: sync pprof@1a94d8640e99
Comment From: gopherbot
Change https://golang.org/cl/260538 mentions this issue: cmd/vendor: sync pprof@v0.0.0-20201007051231-1066cbb265c7
Comment From: gopherbot
Change https://golang.org/cl/266898 mentions this issue: all: update vendored dependencies for Go 1.16 release
Comment From: gopherbot
Change https://golang.org/cl/267103 mentions this issue: all: update dependency on golang.org/x/sys and regenrate Windows syscalls
Comment From: gopherbot
Change https://golang.org/cl/269118 mentions this issue: all: update vendored dependencies for Go 1.16 release (alt)
Comment From: gopherbot
Change https://golang.org/cl/275446 mentions this issue: all: update dependencies with changes in scope for Go 1.16 freeze
Comment From: zikaeroh
@dmitshur Not sure this is the right place to ask, but do you know why pprof is getting skipped over in all of these dependency updates? I submitted a PR there before the freeze and was exited to see my fix show up in the next Go release, but the most recent update pulled an older version before my submission (and it's been skipped since).
Comment From: bcmills
@zikaeroh, our automated scripts only pull in changes to the golang.org/x repos, since those are the only packages that are subject to the Go release cycle.
More generally, though: if you make a change that you want to have incorporated in a Go release (in an x/ repo or otherwise), you should also send a CL to update the go.mod file in the main repo as soon as the fix lands (or is released) upstream.
Comment From: zikaeroh
I see. Is it too late now to update the vendored copy of pprof? I'm happy to submit a CL to do this (if that's easiest).
Comment From: dmitshur
@zikaeroh pprof was updated mostly recently for the Go 1.16 release in ~~CL 248499~~ CL 260538. Are the changes which you'd like to pull in since then in scope of the Go 1.16 code freeze? It's too late for new features and risky changes, but if it's bug fixes then it may be in scope (depending on how risky they are).
Comment From: zikaeroh
It was a fix for google/pprof#515, but I don't know if that qualifies as a bugfix or an enhancement. I can live with building pprof directly and not using go tool pprof if it's too risky.
I was just personally disappointed to see the update CL merged with a version a couple of weeks older than the submission date. I didn't comment at the time as I assumed that CL was a belated pull for the beginning of the cycle, and that there would be another pull for the end of the cycle (which didn't happen).
Note that the most recent pull was actually CL 260538 (which is why I assumed what I did via its commit message). CL 248499 came a few weeks before that.
Comment From: dmitshur
@zikaeroh That bug fix looks reasonable. Please feel free to send a CL updating it (similar to CL 260538), and make me and @hyangah its reviewers. We're still before beta 1, so if the changes look safe to land now, we'll do so (we can revert by beta 2 or RC 1 if it causes significant problems). If the changes look too invasive, it will need to wait for Go 1.17, unfortunately. This is a cost of having a 6 month release cycle—there will be changes that just don't quite make the cut and need to wait.
Comment From: gopherbot
Change https://golang.org/cl/275513 mentions this issue: cmd/vendor: sync pprof@v0.0.0-20201203190320-1bf35d6f28c2
Comment From: dmitshur
By now, the minimum required updates have been done at the end of the Go 1.16 development cycle (via CL 266898 and CL 260538). There are some more updates we need to pull in in order to fix known issues (e.g., issue #31192), but that's tracked by other issues.
Moving this recurring task to the next milestone (Go 1.17) and adding early-in-cycle label.
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.17. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://golang.org/cl/298854 mentions this issue: all: update vendored dependencies for Go 1.17
Comment From: gopherbot
Change https://golang.org/cl/296789 mentions this issue: all: update golang.org/x/* dependencies
Comment From: gopherbot
Change https://golang.org/cl/315831 mentions this issue: all: update vendored dependencies for Go 1.17
Comment From: gopherbot
Change https://golang.org/cl/316110 mentions this issue: cmd: update x/arch to pull in CL 315572
Comment From: gopherbot
Change https://golang.org/cl/316250 mentions this issue: cmd: update x/term to pull in CL 316112
Comment From: gopherbot
Change https://golang.org/cl/316251 mentions this issue: all: update x/net to pull in CL 316129
Comment From: gopherbot
Change https://golang.org/cl/316450 mentions this issue: all: upgrade x/text to its latest release version (v0.3.6)
Comment From: gopherbot
Change https://golang.org/cl/316489 mentions this issue: cmd: update x/mod to pull in CL 316113
Comment From: gopherbot
Change https://golang.org/cl/316490 mentions this issue: cmd: update x/text to pull in CL 315571
Comment From: gopherbot
Change https://golang.org/cl/316451 mentions this issue: all: update x/crypto to pull in CL 316109
Comment From: gopherbot
Change https://golang.org/cl/316452 mentions this issue: cmd: update x/tools to pull in CL 315570
Comment From: toothrot
We may want to change updatestd to create a CL per module to improve bisection results.
Comment From: cagedmantis
This issue is currently labeled as early-in-cycle for Go 1.18. That time is now, so this is a friendly ping so the issue is looked at again.
Comment From: gopherbot
Change https://golang.org/cl/347190 mentions this issue: all: update vendored golang.org/x/crypto
Comment From: gopherbot
Change https://golang.org/cl/347191 mentions this issue: all: update vendored dependencies for Go 1.18
Comment From: gopherbot
Change https://golang.org/cl/347189 mentions this issue: all: update vendored golang.org/x/sys
Comment From: gopherbot
Change https://golang.org/cl/348409 mentions this issue: all: update vendored golang.org/x/tools
Comment From: gopherbot
Change https://golang.org/cl/348410 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: dmitshur
With the 5 CLs above, we're in good shape for early in cycle. Removing that label to revisit this closer to the code freeze.
Comment From: gopherbot
Change https://golang.org/cl/361094 mentions this issue: all: update vendored dependencies for Go 1.18 release
Comment From: gopherbot
Change https://golang.org/cl/361294 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: gopherbot
Change https://golang.org/cl/362734 mentions this issue: all: update vendored golang.org/x/{arch,sys,term} for Go 1.18 release
Comment From: gopherbot
Change https://golang.org/cl/362736 mentions this issue: all: update vendored golang.org/x/crypto for Go 1.18 release
Comment From: gopherbot
Change https://golang.org/cl/362735 mentions this issue: all: update vendored golang.org/x/{net,text} for Go 1.18 release
Comment From: dmitshur
CL 361294 updated github.com/google/pprof. CLs 362734, 362735, 362736, and 361094 updated all golang.org/x dependencies other than x/mod. That is all that's left for the purpose of this issue (other individual updates may also happen, as long as they're bringing in things in scope for the 1.18 freeze).
Comment From: dmitshur
CL 364114 took care of x/mod, the remaining bit for this issue.
Moving to 1.19 early in cycle.
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.19. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://go.dev/cl/393370 mentions this issue: cmd: update vendored pprof
Comment From: gopherbot
Change https://go.dev/cl/393373 mentions this issue: all: update vendored golang.org/x/tools
Comment From: gopherbot
Change https://go.dev/cl/393674 mentions this issue: cmd/updatestd: ignore _ directories
Comment From: gopherbot
Change https://go.dev/cl/393369 mentions this issue: all: update vendored x dependencies for Go 1.19 development cycle
Comment From: gopherbot
Change https://go.dev/cl/406355 mentions this issue: all: update vendored golang.org/x/{arch,sys,term} for Go 1.19 release
Comment From: gopherbot
Change https://go.dev/cl/406356 mentions this issue: all: update vendored golang.org/x dependencies for Go 1.19 release
Comment From: gopherbot
Change https://go.dev/cl/406359 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: dmitshur
CL 406355 and CL 406356 updated golang.org/x dependencies. CL 406359 updated github.com/google/pprof. (Other individual updates may also happen, as long as they're bringing in things in scope for the 1.19 freeze, e.g., to fix release-blocking issues, but that's outside of scope if this issue.)
Moving to 1.20 early in cycle.
Comment From: gopherbot
Change https://go.dev/cl/421334 mentions this issue: all: update vendored golang.org/x dependencies for Go 1.20 development
Comment From: gopherbot
Change https://go.dev/cl/421461 mentions this issue: all: update vendored golang.org/x/{net,tools} for Go 1.20 development
Comment From: dmitshur
CLs 421334 and 421461 took care of all the golang.org/x dependencies, and CL 420234 will update pprof. Once that's done, we can remove the early-in-cycle label and revisit this closer to the freeze.
Comment From: gopherbot
Change https://go.dev/cl/452766 mentions this issue: all: update vendored golang.org/x dependencies for Go 1.20 release
Comment From: gopherbot
Change https://go.dev/cl/452815 mentions this issue: cmd/vendor: update vendored github.com/google/pprof for Go 1.20 release
Comment From: cherrymui
The two CLs above did it for 1.20. Move it to 1.21 early in cycle.
Comment From: gopherbot
Change https://go.dev/cl/462199 mentions this issue: src: update go.mod files to go 1.21
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.21. That time is now, so a friendly reminder to look at it again.
Comment From: dmitshur
CL 467297 and a few others took care of this for early in cycle. Removing that label to revisit this at the start of freeze.
Comment From: gopherbot
Change https://go.dev/cl/498598 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
Change https://go.dev/cl/502878 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
Change https://go.dev/cl/509099 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.22. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://go.dev/cl/517875 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
Change https://go.dev/cl/519657 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: gopherbot
Change https://go.dev/cl/536575 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
Change https://go.dev/cl/546055 mentions this issue: all: update vendored dependencies
Comment From: dmitshur
CL 546055 did the start-of-freeze update for the 1.22 cycle. Moving to the Go 1.23 milestone and adding early-in-cycle.
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.23. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://go.dev/cl/557457 mentions this issue: all: update vendored dependencies
Comment From: cherrymui
CL https://go.dev/cl/557457 submitted for Go 1.23 early-in-cycle. Removed the early-in-cycle label, leave the issue for Go 1.23 end-of-cycle.
Comment From: gopherbot
Change https://go.dev/cl/564636 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: gopherbot
Change https://go.dev/cl/589935 mentions this issue: all: update vendored dependencies
Comment From: mknyszek
CL 589935 did the start-of-freeze update for the 1.23 cycle. Moving to the Go 1.24 milestone and adding early-in-cycle.
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.24. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://go.dev/cl/600535 mentions this issue: all: update vendored dependencies
Comment From: gopherbot
Change https://go.dev/cl/600596 mentions this issue: cmd/pprof: update vendored github.com/google/pprof
Comment From: dmitshur
CL 600535 did the early-in-cycle update for Go 1.24. (Also sent CL 600596 to update a pprof dependency.) Removing that label to revisit this when the freeze starts.
Comment From: gopherbot
Change https://go.dev/cl/623821 mentions this issue: all: update golang.org/x/sys to pull in CL 614082 [generated]
Comment From: gopherbot
Change https://go.dev/cl/631035 mentions this issue: all: update golang.org/x/net [generated]
Comment From: gopherbot
Change https://go.dev/cl/631336 mentions this issue: all: update vendored dependencies [generated]
Comment From: gopherbot
Change https://go.dev/cl/634041 mentions this issue: all: update vendored dependencies [generated]
Comment From: mknyszek
I can't currently make progress on this because of an unexpected new vendored dependency, golang.org/x/sys/unix. This was added in CL 632816 and contradicts an explicit decision in the past to not vendor this package ((#32102). (Thanks goes to @dmitshur for figuring this out and cross-referencing everything.)
We either need to roll back that CL or change the policy. CC @ianlancetaylor
Comment From: mknyszek
Also, I think that this is currently blocking the RC.
Comment From: cherrymui
If we want to unblock the RC, we probably can vendor to the commit right before that CL for the RC? That is probably still recent enough, so we get most of the updates in the x reops?
Comment From: mknyszek
That's fine with me, but do we have tooling to do this? Or would this have to happen manually? AFAICT updatestd doesn't let me actually select a specific commit for different subrepos.
Comment From: dmitshur
Anything other "pick latest versions of all x/ repos" does need some manual intervention. You could still use updatestd, just followed by a go get golang.org/x/net@{pseudo-version before CL 632816}. (Note that updatestd is little more than a convenience command that runs go get ...; go mod tidy; go mod vendor.)
Comment From: mknyszek
I think I am coming to the conclusion that I'd rather roll back the x/net changes. The longer we wait to do so, if we decide not to change the policy, the harder it becomes to revert everything.
Comment From: mknyszek
Reverts landed and I'm unblocked.
Comment From: dmitshur
CL 634041, CL 631336, and CL 626995 took care of the start-of-freeze updates for the Go 1.24 cycle. There may still be more updates to the vendored packages if they're needed to resolve release-blocking issues, but there aren't regularly scheduled updates in scope of this issue left.
@mknyszek Okay to move this to the Go 1.25 milestone, and add early-in-cycle?
Comment From: mknyszek
Yes, I think we're good.
Comment From: gopherbot
Change https://go.dev/cl/643517 mentions this issue: cmd/updatestd: skip repos without internal-branch.go1.n-vendor branch
Comment From: gopherbot
Change https://go.dev/cl/642398 mentions this issue: net/http: update bundled golang.org/x/net/http2 [generated]
Comment From: gopherbot
This issue is currently labeled as early-in-cycle for Go 1.25. That time is now, so a friendly reminder to look at it again.
Comment From: gopherbot
Change https://go.dev/cl/648735 mentions this issue: all: update vendored dependencies [generated]
Comment From: gopherbot
Change https://go.dev/cl/648895 mentions this issue: go/types/typeutil: avoid shiftint uintptr by 32 on 32-bit archs
Comment From: gopherbot
Change https://go.dev/cl/648916 mentions this issue: cmd/pprof: update vendored github.com/google/pprof [generated]
Comment From: dmitshur
With CL 648735, CL 648916, and CL 649296, everything's updated for the early-in-cycle round. Removing that label to revisit this when the release freeze starts.
Comment From: gopherbot
Change https://go.dev/cl/677257 mentions this issue: all: update vendored dependencies [generated]
Comment From: gopherbot
Change https://go.dev/cl/678556 mentions this issue: all: update vendored dependencies [generated]
Comment From: cagedmantis
With CL 678556, and CL 679237, everything's updated. Changing the label to early-in-cycle and the milestone to Go 1.26.