Lots of background and a implementation, albeit from 3+ years ago: https://blog.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/
Basically, Linux now supports handling TLS encryption in the kernel. The primary benefit here is the possibility of sendfile/splice to work with TLS. Currently, we need to choose between TLS and splice (or a custom TLS implementation, I suppose).
It would be great to have first class support in go for this.
Comment From: seankhliao
cc @FiloSottile
Comment From: ShivanshVij
I would love to have this happen as well! It's a major use case for L7 load balancers written in golang, and could transparently provide significant performance boosts for a lot of systems (including Kubernetes)
Comment From: FiloSottile
Can we get some benchmarks and numbers for the performance improvement? My patch linked above might be a good starting point. It's a lot of complexity and it would have to be justified by very good numbers.
Comment From: jim3ma
Hi, all
I have updated kernel tls support based on @FiloSottile's original code. It now supports more ciphers like AES_GCM_256, AES_CCM_128 and CHACHA20_POLY1305.
Code: https://github.com/jim3ma/go/tree/dev.ktls.1.16.3.
And I have fixed some kernel issues when in coding: https://github.com/torvalds/linux/commit/974271e5ed45cfe4daddbeb16224a2156918530e, https://github.com/torvalds/linux/commit/d8654f4f9300e5e7cf8d5e7885978541cf61326b
In my simple tests, when enable kernel tls, I have got 30% time cost decreased.
Comment From: totallyunknown
I made some real-world tests with one of our internal applications (CDN node specialised in delivering video segments for DASH and HLS streams).
- Kernel 5.13.12
- Curve: prime256v1
I compared https vs http, vs http + sendfile and ktls + sendfile.
Most of the TLS stuff is working, except TLS 1.3 with Chrome and k6. k6 reports tls: oversized record received with length 62464.
With ktls, the latency is increased - but this can also be related to the difference in the used Go-Versions.
The ktls implementation reduces overall CPU usage, around 10%. We'll deploy the Nvidia ConnectX-6 (200 Gbit/s) in our latest hardware setup, and we hope we can use the TLS NIC offloading in the future.
https://docs.google.com/spreadsheets/d/1XaiFczae9GLixu__8y2kuKPsw7RGqW9vMDkYxuTLx28/edit#gid=0
Comment From: jrfastab
@totallyunknown If the latency issue is related to the kernel implementation (rule out golang side) we can take a look at kernel side improvements. We've been using the openssl implementation lately so I'll check there as well, but I don't recall extra latency last time I did metrics. Having a golang implementation would be very useful on my side as well. fwiw I'm one of the ktls maintainers on kernel side so we shouldn't have trouble getting improvements there as needed and happy to help where I can to get this moving forward.
Comment From: jim3ma
I made some real-world tests with one of our internal applications (CDN node specialised in delivering video segments for DASH and HLS streams).
- Kernel 5.13.12
- Curve: prime256v1
I compared https vs http, vs http + sendfile and ktls + sendfile.
Most of the TLS stuff is working, except TLS 1.3 with Chrome and k6. k6 reports
tls: oversized record received with length 62464.With ktls, the latency is increased - but this can also be related to the difference in the used Go-Versions.
The ktls implementation reduces overall CPU usage, around 10%. We'll deploy the Nvidia ConnectX-6 (200 Gbit/s) in our latest hardware setup, and we hope we can use the TLS NIC offloading in the future.
https://docs.google.com/spreadsheets/d/1XaiFczae9GLixu__8y2kuKPsw7RGqW9vMDkYxuTLx28/edit#gid=0
Which version do you test ? I have update some go code for http with ktls.
Comment From: totallyunknown
@jim3ma Your branch: https://github.com/jim3ma/go/tree/dev.ktls.1.16.3
Comment From: jim3ma
@jim3ma Your branch: https://github.com/jim3ma/go/tree/dev.ktls.1.16.3
Okay, I will merge some optimized code into this branch tomorrow.
Comment From: kkkygytb
Excuse me, how is the implementation going?
Comment From: kolinfluence
hi, this is such a long awaited feature coz crypto tls is so much slower. pls enable this. thx.
Comment From: VirrageS
@jim3ma are there any plans to introduce the changes into the Go code?
Comment From: jim3ma
@jim3ma are there any plans to introduce the changes into the Go code?
Sorry for busy work. I will rebase kTLS code in latest branch and test it again.
Comment From: zyxkad
any updates?
Comment From: ouvaa
@jim3ma curious about the updates too
been checking here https://github.com/0-haha/gnet-tls-go1-20/ and ref: https://github.com/panjf2000/gnet/issues/534
@FiloSottile i've been watching ktls progress for golang since you started the blog in 2021. this is sort of the final huge golang performance benchmark penalty ever.
once this is ktls-ed, i believe will be one of golang's greatest milestone ever.
Comment From: ShivanshVij
I did some rough benchmarks late last year where I had Golang call into rust's TLS library via CGO to do the handshake and then handed off the established TCP connection to Golang.
I found that the performance (throughput/latency on sustained traffic) ended up being about the same as golang's built-in TLS or slightly worse.
I'm not sure why to be honest - maybe I did something wrong? But I would like to see some numbers hopefully from someone else on the actual performance of the kTLS implementation in the linux kernel.
Comment From: ouvaa
@ShivanshVij u hv the code for helping to debug? but ktls is better for sure.
Comment From: zyxkad
I did some rough benchmarks late last year where I had Golang call into rust's TLS library via CGO to do the handshake and then handed off the established TCP connection to Golang.
Based on my understanding, kTLS does not magically works, it's used for zero copy, so you have to send a fd through syscall
Comment From: ShivanshVij
Yep - so the implementation was really straight forward.
Start a TCP listener, wait for a TCP connection to get accepted, and read some N bytes from it and send them to rustls via CGO. If we needed more bytes the rustls library would signal that, otherwise it would give us some bytes to write back to the connection - which we would do in Go by blindly writing the byte slice into the net.Conn.
Once the handshake was complete, we'd pull out the required kTLS secrets from the handshake in rustls, and then do the required syscalls in Go to tell the kernel that the fd that backed the net.Conn is a kTLS fd.
After that, future reads/writes on the net.Conn would result in proper TLS encryption/decryption without any userspace overhead.
Comment From: kanocz
One more thing - many better network card have crypto-acceleration and this can be accessed by ktls API, so supporting ktls in golang we are able to offload encryption to network card so please don't compare only software encryption in golang vs software encryption in kernel - it's not so relevant for many production environments
Comment From: kolinfluence
i've used gnet's ktls and other ktls version but i found that if going through cgo, and with multiple goroutines, it seems to crash. e.g. 1000000 goroutines calling cgo seemed not possible. u can probably do 80k max. so not sure if ktls will be available to do so or if this will be an issue crashing if doing cgo syscall etc.
Comment From: rsc
This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group
Comment From: rhysh
It looks like rustls (for Rust) makes kTLS possible by allowing access to the key material after the handshake completes. Could that be the right stance for Go as well, to allow use of kTLS without the crypto/tls maintainers needing to take on ownership of all of the moving parts?
The QUIC support in crypto/tls is in a similar position, where crypto/tls does the initial handshake and then hands the key material over to its caller.
From what I can tell, the discussion at https://github.com/rustls/rustls/issues/198 led to https://docs.rs/rustls/latest/rustls/struct.ExtractedSecrets.html, which in turn enables users to provide their own kTLS wiring. (There's support in crypto/tls already for a Config.KeyLogWriter — but as the rustls maintainers also discovered, that format doesn't include all of the information that the kernel needs to continue the symmetric encryption.)
Comment From: totallyunknown
We aim to achieve 400Gbit/s network throughput serving HTTP with Go but are currently constrained by memory bandwidth. With AMD Rome generation, we can reach 165Gbit/s of network traffic, with memory bandwidth fully utilized, as shown by AMDuProf. To overcome this, we need zero-copy techniques like sendfile, which requires kTLS support in Go, eliminating the memory bandwidth constraint.
Comment From: rsc
@rolandshoemaker and @FiloSottile to work out an API. It sounds like we should work on an API where Go keeps the handshake and then hands off the key so the kernel can do the record layer.
Comment From: rsc
@FiloSottile and I discussed this, and we wonder if this can be done without any new secret-sharing API at all: if kTLS is good enough, then Go should arrange to use it by default, right? We'd probably also need to add ReadFrom and WriteTo methods to the tls.Conn implementations so that io.Copy goes straight to sendfile, but no new TLS-related API would be needed.
Is there a flaw in this thinking?
Are there Go or Rust kTLS implementations already that are worth looking at to understand the kernel interaction details? We spent a while reading linux/tls.h but it's not terribly well documented.
And are there other operating systems with kTLS that we should look at?
Comment From: 4xoc
I believe it would the be the right thing to get kTLS going as a default on supported systems. Having a secret sharing API might be useful for some developers though, maybe something one can meddle with explicitly.
Maybe this helps with the kernel interaction.
Looking at FreeBSD would probably be a good idea. The implementation seems quite mature.
Comment From: astrolox
Nginx has had support since around 2021. Although I think it just delegates the hard work to OpenSSL. Still might be worth a look here; https://hg.nginx.org/nginx/rev/65946a191197
Comment From: rsc
I believe it would the be the right thing to get kTLS going as a default on supported systems. Having a secret sharing API might be useful for some developers though, maybe something one can meddle with explicitly.
Meddlers can always use reflect and unsafe. No need to add API for them.
Comment From: sprappcom
here, some of the unverified and broken ktls on my radar: https://github.com/0-haha/gnet-tls-go1-20/blob/dev/ktls_linux.go https://github.com/soluble-ai/go-ktls/blob/master/ktls.go
when's the eta for this? been looking at this thread since 2021. :D
@totallyunknown 's doing 165GBits/s on a 400Gbits/s line is really weak. i'm hoping for the performance too.
@rsc possible for meddlers to live with one without the alloc/op too? that'll be heaven.
talk about zero alloc/op... i really wish arena feature is fully supported as non-experimental.
Comment From: harshavardhana
https://github.com/soluble-ai/go-ktls/blob/master/ktls.go
This is not kernel TLS. It looks like some TLS secret as Kubernetes secret
Comment From: rolandshoemaker
Possibly on the roadmap for 1.24 if we have the time.
Comment From: totallyunknown
@totallyunknown 's doing 165GBits/s on a 400Gbits/s line is really weak. i'm hoping for the performance too.
@sprappcom 400G is the future goal. 165 Gbit/s is with 2x100G (NVIDIA Connect-X6 + AMD Rome).
Comment From: sprappcom
@totallyunknown ok. 82.5% is impressive. mine can only do 60% on laptop
Comment From: rsc
It sounds like people are on board for "no new API", implementation on by default once it works, with a GODEBUG like tlskernel=0 to turn off.
Do I have that right?
Comment From: howardjohn
I have a few concerns about on-by-default:
There is a meaningful difference in data being written to the kernel in plaintext vs encrypted, from a debugging, tooling, and even security POV. (I would not claiming there are legitimate threat vectors here, but some people are quite paranoid, and I am not an expert -- so I suspect others might).
kTLS may work on a wide-ish range of Linux versions which we can check against, but it doesn't necessarily work well on all of them. https://people.kernel.org/kuba/tls-1-3-rx-improvements-in-linux-5-20 for instance shows there are some very recent critical performance improvements. This makes it tricky to know what is the right bar to implicitly turn this on. Is it the oldest Linux version that supports the features indicated? The oldest one we measured as "fast enough"? What if changes to Go or Linux change whether it is "fast enough", or different kernel or hardware configurations do? For instance, kTLS may be "not fast enough" on Linux 5.0 in general, but I may have a NIC that supports kTLS offload making it suitable even on that version.
I won't claim to be a TLS expert, it just feels like there are a tremendous number of variables to consider. I could see maybe in many years there is enough clarity on real world use cases, production testing, etc that we get to the point where we could turn it on by default. However, I don't think that would happen for many years likely, and even when it did I would think it is controversial enough to warrant first class configuration rather than just a GODEBUG.
FWIW, around a year ago I did some performance testing of Go+kTLS with some fork I found (sorry, I forget a lot of details at this point). The performance was pretty rough, and, surprisingly, even worse when using splice which is what should be the big win. I wouldn't put too much weight on that given the vague claims + age + unofficial implementation, but something to keep an eye on.
Comment From: Jorropo
@howardjohn about the point of which is the first good enough version.
I don't see why we shouldn't default to using it with a linux version where linux's software implementation is good enough to match / beat go's one because at worst performance is similar but you can now sendfile and friends.
5.0 being slow is not a good reason to arbitrary slow down 5.20 (or whatever the threshold is).
We can make a ternary option with GODEBUG=tlskernel=always for the users on 5.0 that happen to have compatible hardware.
We already conditionally use linux features based on the kernel's version in the codebase: https://github.com/golang/go/blob/fed2c11d67dbe6d8179cd411b4bb7761d034e9d2/src/internal/poll/copy_file_range_linux.go#L18
These old benchmarks showed worst latency even when throughput is significantly higher. However this is already a problem in the TLS protocol due to AEAD framing tradeoffs and here is our strategy against it: https://github.com/golang/go/blob/52ce25b44e8c21f62e95b12497db3036c5bd27c3/src/crypto/tls/conn.go#L879-L894 If this happens again because handshakes will still happen in userland go we don't need to transition right away to KTLS, we can handle the first 1MB (or Xbytes) in userland to keep a healthy time-to-first-byte.
Comment From: FiloSottile
The fact that deciding when to enable kTLS is hard is not a reason to delegate the choice to the user, but the opposite: it means we should do the research, measure the performance, weight the tradeoffs, and make the judgement calls, so our users don't have to. We're in the business of building a TLS stack, our users are in the business of writing Go programs.
More concretely, yes, I think we should figure out which Linux version has "good enough" kTLS, and require that. Users that want kTLS can upgrade their kernel version, and as time passes that will be less and less of a problem. I am not too concerned about weird combinations of old kernels and powerful NICs, if they are common, we'll hear about it and reassess.
Comment From: sprappcom
ktls are for extreme users, just go extreme in this order (seriously, i doubt there will be more than 100 proj adopting this coz i'm looking at this thread since 2021) 1. "future proof" linux compat (coz i'm into linux and the userbase for this is the largest and of course most important) 2. security (i thought it's just api so no need to worry that much about this ktls security coz linux side would have taken care of it. just allow us to upgrade to the latest kernel will do)
get it out already by now, pls at go 1.24. make it an experimental feature at least. thx in advance. this is really tough to get right i understand.
@rsc @FiloSottile GOEXPERIMENT=ktls
those who want edge cases can fork it and do the package they want on their own os.
Comment From: rsc
Let's keep the GODEBUG name starting with tls like all the other tls names, so tlskernel=1 not ktls=1. GODEBUG not GOEXPERIMENT because it is a runtime decision.
It sounds like we all agree not to add new API other than the GODEBUG. With the proposal being just to add the GODEBUG and work on an implementation, I think this is moving towards acceptance. Do I have that right?
Comment From: rsc
Have all remaining concerns about this proposal been addressed?
The proposal is to develop transparent kTLS support behind GODEBUG=tlskernel=1. A future proposal can discuss the conditions under which it should be enabled by default.
Comment From: scwgoire
Hi guys, I have a use case you may want to hear. Dropping it here even though I'm not 100% sure kTLS would be the solution, my TLS knowledge is very limited, sorry about that.
My use case is seamless application upgrade.
My TCP clients use long-lived (weeks or more) connections and allowing software updates without disconnecting them would be a killer. The application is not very complex, I can already serialize its internal state and pass it with the open sockets to the upgraded instance of the software. However when TLS joins the game, it gets tougher, using kTLS may eliminate this problem (considering there is nothing more to do at application level once the kernel takes over the socket, which I honestly don't know).
I understand this could very well be achieved without kTLS with lots of knowledge and hacking, but I don't want to do it the hacky way and my readings make me think this use case is not supported by the current crypto/tls API and there is no plan to do so. Am I right?
Client TLS session resumption is not an option unfortunately, I have no control on the many possible client software.
Hope this helps 😃
Comment From: ShivanshVij
I wonder if there would be an issue passing the open socket to a separate process if kTLS is involved... my initial research says there wouldn't be, but definitely worth testing.
Comment From: astrolox
I hadn't even imagined passing a TLS enabled socket to another process. That would definitely be useful for me also. Currently we use a separate process to handle TLS and then proxy the connection over unix sockets to another process does the actual work. This causes unwanted resource usage and latency, but allows us to keep the connections open while recycling the worker process. If kTLS could make this whole design a lot simpler, we'd be very happy campers.
Comment From: kajtzu
Not giving you a complete solution but do take a look at pidfd_open(2), pidfd_getfd(2), pidfd_send_signal(2), @astrolox, if you wish to avoid unix socket with recent (5.6?, 5.10?) linux kernels.
Comment From: scwgoire
I wonder if there would be an issue passing the open socket to a separate process if kTLS is involved... my initial research says there wouldn't be, but definitely worth testing.
I got a proof-of-concept working here. I basically adapted the first attempt from @FiloSottile to up to date Go TLS stack. I also added support for enabling TLS_RX on the socket. (example from Filippo only configured TLS_TX)
In the new process, you end up with a net.Conn in which you need to: - write TLS record header + plaintext data - read plaintext data only
I still need to have this run for a long time with lots of data to make sure there is no catch.
Potential drawbacks with the new process: - can't handle incoming non-data records - can't issue non-data records (the Conn is a net.Conn, not a tls.Conn)
Also, it need support for all ciphers, only implemented it in prefixNonceAEAD
Comment From: rsc
Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group
The proposal is to develop transparent kTLS support behind GODEBUG=tlskernel=1. A future proposal can discuss the conditions under which it should be enabled by default.
Comment From: rsc
No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group
The proposal is to develop transparent kTLS support behind GODEBUG=tlskernel=1. A future proposal can discuss the conditions under which it should be enabled by default.
Comment From: ghost
I have an implementation of Go kernel tls. Its patch standard library but can directly use import. More features like TLS compress certificate extension and more.
For master, fork from Go v1.22 with cloudflare branch ECH implement Go v1.23 patches in dev branch. Its will be released after Go standard library ECH server side implement (maybe 1.24).
See Repository
You can use it as a reference.
Comment From: ghost
One more thing.
Here is test result with iperf3 using this package. - TLS 1.3 CHACHA20 - Linux 6.13 (splice) SW (4.69 Gbps) > kTLS TX (4.65 Gbps) > kTLS RX (3.98 Gbps) > kTLS Both (3.88 Gbps) - FreeBSD 14 (no splice) SW (3.05 Gbps) > kTLS RX (1.89 Gbps) > kTLS TX (1.71 Gbps) > kTLS Both (935 Mbps)
It seems that throughput is not as good as software implementation, even with zerocopy.
It is recommended that the option be given to the user rather than being turned on by default.
Comment From: 0-haha
The fact that deciding when to enable kTLS is hard is not a reason to delegate the choice to the user, but the opposite: it means we should do the research, measure the performance, weight the tradeoffs, and make the judgement calls, so our users don't have to. We're in the business of building a TLS stack, our users are in the business of writing Go programs.
More concretely, yes, I think we should figure out which Linux version has "good enough" kTLS, and require that. Users that want kTLS can upgrade their kernel version, and as time passes that will be less and less of a problem. I am not too concerned about weird combinations of old kernels and powerful NICs, if they are common, we'll hear about it and reassess.
I did a research on KTLS kernel support matrix
| 4.13 | 4.17 | 5.1 | 5.11 | 5.19 | 6.0 | |
|---|---|---|---|---|---|---|
| TLS 1.2 TX | X | x | x | x | x | x |
| TLS 1.2 RX | x | x | x | x | x | |
| TLS 1.3 TX | x | x | x | x | ||
| TLS 1.3 RX | x | |||||
| AES-GCM-128 | x | x | x | x | x | x |
| AES-GCM-256 | x | x | x | x | ||
| CHACHA20POLY1305 | x | x | x | |||
| TLS zerocopy | x | x | ||||
| TLS nopad | x |
I think kernel >= 6.0 is a good choice to support all necessary features. From 6.1, ktl supports aria-gcm, which is not supported in go tls. So kernel 6.0 should be fine.
I implemented one in https://github.com/secure-for-ai/goktls two years ago. The library enables the ktls feature based on the support matrix. You can find a working example at https://github.com/secure-for-ai/goktls_examples. Recently, I tried to patch it from go 1.20 to 1.24. Howevr, it take a lot of work to patch it since the latest tls lib uses a lot of internal packages. It would be nice if the upstream can natively support the ktls.
The implementation idea is to let the go tls library to handle the handshake. Kernel tls is enabled based on the kernel version and the ciphersuite. Once enabled, the read and write are offloaded to the kernel. Only TX may be offloaded to Kernel tls as it is shown in the supporting matix.
Comment From: howardjohn
It's not just about the feature matrix - the performance and security properties of ktls are quite different than software. I get that Go has a (pretty reasonable) minimal-config philosophy for TLS, but I don't think extending that to enabling kTLS by default is good for anyone. The risk of enabling kTLS at this point is substantial - defaulting every Go user into that would be a huge risk for thousands of projects.
If the proposal is to keep it under a permanent GODEBUG opt-in that seems to meet that goal, I suppose, though isn't a great UX for users that do want to opt in.
Comment From: FiloSottile
The risk of enabling kTLS at this point is substantial
Can you elaborate, to help us make that risk assessment?
Comment From: howardjohn
The risk of enabling kTLS at this point is substantial
Can you elaborate, to help us make that risk assessment?
Go TLS stack is used by a huge number of projects, orders of magnitude greater than kTLS - and its the same for everyone. kTLS is relatively immature and it isn't just "one thing" but rather delegated to the whims of the host we happen to run the machine on. https://github.com/golang/go/issues/44506#issuecomment-2604398555 shows a huge range of change across Linux versions, and there are more that are less visible changes happening all the time that can have huge (positive or negative, presumably mostly positive though) performance implications. We may test on one machine and find the results are positive and decide to roll it out by default, only for 100s of other machines to experience severe regressions upon upgrading Go.
Flexibility is a huge issue as well. The Go team can change crypto/tls at any time to improve perf/fix bugs/add features/etc -- not so with kTLS. For example, consider some new TLS CVE coming out that requires a patch to the handshake logic. With Go currently, just a Go patch goes out. With kTLS, either we flip the default from kTLS -> Go TLS in a patch release (and then probably flip it back later!) which is rough, or we tell users "go update your kernel" which is usually not great.
Performance is also widely variable. There is a pretty big gap in performance analysis of kTLS in diverse scenarios - I would imagine that is a pre-req to this being made default, though, which would help there. But anecdotally, it seems the primary big users of kTLS + the research papers/benchmarks are all around sendfile, while other paths see regressions (see https://github.com/golang/go/issues/44506#issuecomment-2387977030 and others). I would also note that I had also seen terrible results with splice, which would be used often automatically with io.Copy, though I haven't tested since ~Linux 6.0.
The side of me that likes to explore new technology and see where we can optimize every last bit of performance is excited for the prospect of using kTLS. However, the other side of me that maintains a stable Go project depended on by many is terrified at the prospect of being forced to use it.
Comment From: francislavoie
Caddy maintainer here, I would agree that on-by-default is not correct due to the aforementioned risks. I think users trust Caddy partly due to knowing it uses the Go TLS stack (memory safe etc, is not OpenSSL), so silently using another TLS implementation would be a problem.
I would hope that it can be enabled via tls.Config (I think?) so users can opt in via config if they want to try/use kTLS.
Comment From: dfinkel
Last I checked, kTLS doesn't do TLS handshaking, so the riskiest parsing, allocation, etc. would still be using Go's TLS implementation.
Given that the current proposal only specifies using GODEBUG, it sounds like the plan has always been to leave kTLS support behind GODEBUG until it's deemed stable enough for some set of kernel versions before flipping it to be the default under whatever circumstances. (with some opt-out)
I feel like arguing against default-enabling based on the current state of the kernel is putting the cart before the horse.
If people can easily opt-in, and we start exercising the splice code paths that were relatively unoptimized before, then there will be incentive (and metrics) to support kernel work to fix any bugs/performance regressions, etc.
I'm assuming that there will be a bunch of fuzzing, before anyone has the confidence to flip that default.
I wouldn't be surprised if an eventual minimum kernel version to default-enable kTLS hasn't come out yet.
Comment From: 0-haha
Last I checked, kTLS doesn't do TLS handshaking, so the riskiest parsing, allocation, etc. would still be using Go's TLS implementation.
Yes, KTLS only handles read and write the socket. TLS handshake leaves to Go implementation. Depending on the kernel version and the ciphersuite, kTLS can be enabled only for writing but not reading, or completely disabled.
I get that Go has a (pretty reasonable) minimal-config philosophy for TLS, but I don't think extending that to enabling kTLS by default is good for anyone.
I agree. If kTLS can be implemented in go as an experimental feature, at least it can gather the input from the user without breaking existing projects. You would never know how to optimize/fix the code until you get the data from a real use case. Linux kernel started to have kTLS from 4.13 and didn't finish most of the features until 6.0. It can take years for kTLS getting mature. But the point is that you cannot make any progress until you start it.
Using GODEBUG is like to have a safe guard as the kernel can change the implementation of kTLS at any time. This leaves the user an option to switch back instantly without an issue.
In addition, kTLS is not enabled on modern linux by default. You have to run sudo modprobe tls to enable it on ubuntu. Maybe only serious users would try it. To test the performance, you need 100G network. I guess only some big players have the inventive to try this, like CDN operators who care about the throughput and the CPU usage.
Comment From: rbqvq
https://gitlab.com/go-extension/tls for Go v1.24 released, I forgot to inform here.
For those interested, you can replace import crypto/tls to this library.
It is fully standard library compatible.
This repository provides some advanced functions, and since we need to keep up with the upstream, its API may not be so stable if the upstream conflicts with our changes.
You only need to set KernelTX and KernelRX to true in tls.Config after changing the import.
In addition, kTLS has some problems, for example, sending some BADMSG (e.g. bad tls version) after splice will cause io.Copy to be stuck.
Entering splice on FreeBSD may not check the TLS message version number. This may be a security risk.
That's all I can remember, if you're interested in testing it yourself, send in some different BADMSG. Enabling it for production environments can cause accidents.
For me, I enable kTLS offload in the TX direction only
The above conclusions are based on tests on Linux 6.13 and FreeBSD 14.
Expect someone to test the new version of the Linux and FreeBSD.
kTLS is an expert feature, and because it is bundled with the kernel, it works differently in different environments, so we recommend giving the option to the user and turning it off by default. If the user is a TLS expert, the user will see this. Conduct appropriate benchmarks to ensure that there is no performance rollback.
So we don't need GODEBUG either. Because it is off by default
This repository will continue to be updated every time a major version of Go is updated, and if I need help, I can consider making a patch to the Go standard library.
Comment From: lwwgo
Golang is very good code language. I’ve been following this issue closely and check in every few months. I was excited when it was accepted last year, but it's been a year now and there still doesn’t appear to be any concrete implementation plan from the Go team. This feature is highly important for our HTTP service. Could it be prioritized and assigned to a specific milestone, such as Go 1.26, if possible?
Comment From: szuecs
Caddy maintainer here, I would agree that on-by-default is not correct due to the aforementioned risks. I think users trust Caddy partly due to knowing it uses the Go TLS stack (memory safe etc, is not OpenSSL), so silently using another TLS implementation would be a problem.
I would hope that it can be enabled via
tls.Config(I think?) so users can opt in via config if they want to try/use kTLS.
Skipper maintainer here and I fully agree with this statement.
Comment From: rbqvq
Caddy maintainer here, I would agree that on-by-default is not correct due to the aforementioned risks. I think users trust Caddy partly due to knowing it uses the Go TLS stack (memory safe etc, is not OpenSSL), so silently using another TLS implementation would be a problem. I would hope that it can be enabled via
tls.Config(I think?) so users can opt in via config if they want to try/use kTLS.Skipper maintainer here and I fully agree with this statement.
I agree.
I think, for now, I am pessimistic about ktls
My repository gitlab.com/go-extension/tls implements most of the things that the standard library doesn't have, but its API is not stable
I can't be sure that the standard library will not add the same fields as me, which has already happened once (This repository aims to ensure full 1:1 compatibility with the official library, so our API will change when there is a conflict)
It still has many problems
On FreeBSD, my repository cannot implement acceleration for TLS 1.0 CBC, it will cause BADMSG (although it is a TODO, I think I will never implement it, these cipher suites are almost not used)
The remaining known issues can be found here https://github.com/golang/go/issues/44506#issuecomment-2765047544
My performance test https://github.com/golang/go/issues/44506#issuecomment-2387977030
I have proposed to quic-go and caddy to add a fork of ktls, But it was not accepted
My suggestion is to use the ktls build flag go build -tags ktls
and redirect all the content that uses tls to the internal package
to ensure easy switching of the tls implementation backend
The default compilation is still the standard library implementation, which will be a better choice for users who are willing to take risks and try new things
For existing libraries, if a large number of standard libraries are imported, this will be a big change
For my own settings, I only turn on TX offload. At present, there are too many problems with RX, and the performance regression is serious
Anyway, you can use my repository to perform benchmarks and other tests yourself
Since go.mod replace does not support replacing standard libraries, in addition to patching crypto/tls, the best option is to redirect all used tls types, constants, variables, and functions to internal packages
In this way, we only need to copy the standard library and modify the import once to switch to any other tls backend
Here is a reference https://gitlab.com/go-extension/quic-go/-/tree/master/internal/qtls?ref_type=heads
crypto/tls uses pointers rather than interfaces extensively. I think that in the future, if crypto/tls/v2 is released, it should consider the features of custom backends.
Comment From: rbqvq
I noticed that Linux 6.14 has added a rekey method to support KeyUpdate.
Actually, there's a problem.
We have no way of knowing if the kernel setup will fail.
If the setting is successful and then the KeyUpdate is replied, the decryption fails because the peer does not change the key
If you reply to KeyUpdate in advance, the Alert sent by us cannot be decrypted by the peer after the kernel setting fails.
Are there experts on (K)TLS?
Is there really a client/server that sends a KeyUpdate?
The current solution implemented in the gitlab.com/go-extension/tls is to check the pre-condition in advance.
Returns a function or error
If the precondition check is not passed
An InternalServerAlert is sent immediately
If the precondition check passes, Reply KeyUpdate to peer, then call the returned function to set the new key. If this function call fails, nothing is sent, but an error is directly set, waiting for the user to manually close the connection.
Commit can be found in https://gitlab.com/go-extension/tls/-/commit/c7de9f28f4939bd2996ce9bd6ddde4bf51174088
Let me know if there is a better suggestion.