Golang Thread and Memory Leak in Kubelet and Go Program with Multiple Goroutines

Go version

go1.20.5 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nec/.cache/go-build"
GOENV="/home/nec/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/nec/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/nec/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3908633320=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I’ve observed a persistent issue with thread and memory retention in both the Kubernetes Kubelet and a custom Go program that utilizes multiple goroutines. After workloads are terminated, the Go runtime appears to retain OS threads and memory, which aren’t released as expected, leading to elevated thread counts and memory usage over time.

Steps to Reproduce:

1. Kubelet:

2. Deploy a Kubernetes cluster with Kubelet running as a pod.

3. Apply a workload that creates high CPU and memory pressure (e.g., a deployment with containers running frequent readiness probes).
4. Monitor the number of threads and memory usage of the Kubelet process before, during, and after the workload.
5. Observe that threads and memory are not released even after the workload is terminated.

Custom Go Program: - Write a Go program that spawns a large number of goroutines, each performing some work (e.g., simple computations or I/O tasks). - Monitor the thread count and memory usage after the goroutines complete. - Observe that the OS threads and memory usage remain elevated.

I have experimented with adjusting the following Go runtime environment variables in the kubelet config file: - GOMAXPROCS: Set to different values to control the number of CPU cores. - GOMEMLIMIT: Set to a low value to force aggressive memory reclamation. - GOGC: Set to a low value (e.g., 50) to increase the frequency of garbage collection.

However, these adjustments did not affected the thread and memory.

Is this behavior expected due to some inherent Go runtime behavior, or could this be an actual bug in the runtime’s thread and memory management? Any guidance on further debugging steps or potential workarounds would be greatly appreciated.

What did you see happen?

Kubelet: After workload termination, the Kubelet process retains a high number of OS threads and does not release the associated memory, resulting in persistent elevated resource usage.

Go Program: Similarly, in the custom Go program, OS threads and memory are not released after the goroutines have completed, leading to the same issue of resource retention.

What did you expect to see?

In both cases, I expected that once the workload (in Kubelet) or the goroutines (in the Go program) were completed, the Go runtime would release the idle OS threads and free up the allocated memory, returning the process to its baseline resource usage.

Comment From: gabyhelp

Related Issues and Documentation

• runtime: Go routine and writer memory not being released #32124 (closed)
• runtime.malg: memory leak #66564 (closed)
• runtime: threads and their Ms aren't released promptly enough in some case #56668
• runtime: memory of finished goroutines not freed #65864 (closed)
• runtime: Kubernetes' kube-proxy stuck in GC forever #18925 (closed)
• go http concurrency and Goroutines do not release memory!!! #15590 (closed)
• runtime: 1.3 garbage collector not releasing server memory back to system from finished goroutines #8287 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: ianlancetaylor

Yes, this is current expected behavior. Closing this issue as a dup of #14592. Thanks.

Comment From: prattmic

Note that Go does not hold onto all memory forever (though it will keep the thread stacks since it doesn't exit the threads).

Memory freed by the GC isn't immediately returned to the OS (so it can be reused quickly), but the runtime does release excessive memory to the OS eventually. The amount of memory to release is informed by GOGC and GOMEMLIMIT. If GOMEMLIMIT seems to have no effect, then it may be worth looking at the /gc and /memory metrics in runtime/metrics to see where the memory is going.

cc @mknyszek