Go version
1.24.2
Output of go env in your module/workspace:
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/yutaro.linux/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/yutaro.linux/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2500888817=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/yutaro.linux/go/1.24.2/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/yutaro.linux/go/1.24.2'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/yutaro.linux/.goenv/versions/1.24.2'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/yutaro.linux/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/yutaro.linux/.goenv/versions/1.24.2/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.24.2'
GOWORK=''
PKG_CONFIG='pkg-config'
What did you do?
func TCPMD5SigAvailable() (bool, error) {
listener, err := net.Listen("tcp", "localhost:0")
if err != nil {
return false, fmt.Errorf("listen failed: %w", err)
}
defer listener.Close()
fd, err := listener.(*net.TCPListener).File()
if err != nil {
return false, fmt.Errorf("error retrieving socket's file descriptor: %w", err)
}
defer fd.Close()
err = unix.SetsockoptTCPMD5Sig(int(fd.Fd()), unix.IPPROTO_TCP, unix.TCP_MD5SIG, newTcpMD5Sig("1.2.3.4", "key"))
if err != nil {
if errors.Is(err, syscall.ENOPROTOOPT) {
return false, nil // "protocol not available"
}
return false, fmt.Errorf("other error by setting setting socket option: %w", err)
}
return true, nil
}
func newTcpMD5Sig(address, key string) *unix.TCPMD5Sig {
sig := &unix.TCPMD5Sig{}
addr := net.ParseIP(address)
if addr.To4() != nil {
sig.Addr.Family = unix.AF_INET
copy(sig.Addr.Data[2:], addr.To4())
} else {
sig.Addr.Family = unix.AF_INET6
copy(sig.Addr.Data[6:], addr.To16())
}
sig.Keylen = uint16(len(key))
copy(sig.Key[0:], key)
return sig
}
func main() {
available, err := TCPMD5SigAvailable()
if err != nil {
fmt.Printf("Error checking TCP_MD5SIG availability: %v\n", err)
return
}
if available {
fmt.Println("TCP_MD5SIG is available")
} else {
fmt.Println("TCP_MD5SIG is not available")
}
}
Execute above code with sysctl net.mptcp.enabled=0 (MPTCP disabled) and net.mptcp.enabled=1 (MPTCP enabled) on the Linux with CONFIG_TCP_MD5SIG enabled. With 1 it shows TCP_MD5SIG is not available and with 0, it shows TCP_MD5SIG is available.
What did you see happen?
Since v1.24, net.Listen started to enable MPTCP by default for all TCP listeners on Linux.
This makes kernel to call setsockopt of MPTCP (mptcp_setsockopt) instead of TCP.
For some options are compatible with TCP, so they are handled transparently, but in case of MD5, it is not supported in MPTCP, so it returns ENOPROTOOPT here.
This breaks the existing program that uses net.Listen with MD5 digest. The concrete example is BGP speaker such as gobgp. The issue was originally reported for Cilium which internally uses GoBGP (ref).
I'm not entirely sure how we should solve this problem. In theory, this problem occurs for any socket option that is supported in TCP, but not supported in the MPTCP. So far, MD5 digest and TCP_REPAIR are not supported. The former is used in BGP and the latter is used in the container migration tool like CRIU (this is not written in Go though).
~~The workaround we can make is setting multipathtcp GODEBUG or disable MPTCP with either sysctl or kernel configuration. Both doesn't allow us to adjust the behavior per socket. Since the error happens after socket creation, falling back to the TCP is pretty hard at that point.~~
(I was overlooking the net.ListenConfig.SetMultipathTCP)
What did you expect to see?
The existing program works without breakage.
Comment From: gabyhelp
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
Comment From: ianlancetaylor
CC @matttbe
Comment From: matttbe
Hello,
@YutaroHayakawa thank you for having reported this issue!
@ianlancetaylor thank you for the cc!
In short, TCP MD5 is not compatible with MPTCP, because the MD5 digest in the TCP options is taking 18 bytes + 2 bytes for the padding (if I remember well), out of the 40 bytes in total. There is space left is not enough for MPTCP which typically requires 20 to 28 bytes (depending on the negotiated DSS size) when carrying data. (And by default, TCP Timestamps is used which is 10+2 bytes + eventually SACKS...). For this reason, setsockopt(TCP_MD5SIG) on an MPTCP socket is not supported in the Linux kernel.
But... here we are in a different situation: MPTCP is used by default, and I guess with BGP, the client will not request MPTCP. What we could do, is to support TCP_MD5SIG with an MPTCP socket, which will somehow disable MPTCP support if requested. Would that be a more convenient solution here?
I'm not sure what we could improve in net in Go because, if I remember well, the stream socket (when we decide to use MPTCP) is created when calling net.Listen(), and it is only after that MD5 is requested. @YutaroHayakawa's fix to disable MPTCP in gobgp seems to be the good choice. I guess nobody else is using TCP MD5, it was design for BGP if I'm not mistaken. (and maybe the kernel "fix" is not really needed, but it might be better)
Comment From: YutaroHayakawa
While this is a breaking change, I don't think there's a significant impact TBH. As you mentioned, MD5 is for BGP and BGP speaker written in Go is already rare (GoBGP is the most widely used one which is already fixed), the users rarely use MD5 feature, and there's a direct successor TCP-AO (not sure if MPTCP supports that though). I don't think we need to fix the kernel or Go standard library side just for this use case. BGP side can deal with it. It was a single line fix for GoBGP.
Comment From: cherrymui
cc @neild
Comment From: seankhliao
Sounds like we're not going to make any changes to net? Perhaps a release note addition to 1.24?
Comment From: matttbe
Sounds like we're not going to make any changes to net?
Out of curiosity, are you also using TCP MD5? In which context?
I'm asking this, because we are working on a patch for the kernel side, but it is not clear what is the impact of not supporting this socket option with MPTCP: TCP MD5 is very niche, and, in our mind, only used by BGP. If it is more common than that, we should make sure this fix is backported.
Comment From: seankhliao
I'm not using it, I'm just going through open issue
Comment From: neild
To check my understanding: As of Go 1.24, net.Listen enables MPTCP by default on Linux. Some socket options are not compatible with MPTCP, and programs which use those socket options are broken by this change.
Possible fixes include:
- Revert the change to enable MPTCP by default. It sounds like nobody thinks this is a good option, since usage of the affected socket options is extremely rare and specialized, and we have encountered no other problems related to MPTCP in the time since 1.24 was released.
- Disable MPTCP automatically if an MPTCP-incompatible option is used. I don't think this is feasible, since MPTCP is selected at socket creation time.
- Do nothing and require that programs which use MPTCP-incompatible options explicitly disable MPTCP with
ListenConfig.SetMultipathTCP(false).
In the absence of some better suggestion, I think we're going to have to go with option 3 (do nothing). It's unfortunate that this change broke any programs (we try very hard to avoid backwards incompatible changes), but the number of affected programs is small (possibly just GoBGP?), the fix is straightforward and simple, GoBGP has already put in that fix, and reverting the change in 1.25 would just be more churn with little benefit.