Similar https://github.com/awslabs/s2n/issues/607, Having access to raw ClientHello can be useful for fingerprinting clients [1] for further analysis. Plus, With raw ClientHello message, we could also implements SNI Proxy in tls.Config.GetConfigForClient [2] , e.g. tlsrouter [3] more easily. In openssl this can be done by setting up callback through SSL_CTX_set_msg_callback. Would be nice to have similar ability for golang crypto.
[1] https://github.com/salesforce/ja3 [2] https://golang.org/pkg/crypto/tls/#Config [3] https://github.com/google/tcpproxy/tree/master/cmd/tlsrouter
Comment From: bcmills
CC @FiloSottile
Comment From: deancn
mark
Comment From: mysticaltech
That is really needed!
Comment From: deancn
Possibly add a raw byte for TLS?
Just reference: https://github.com/bfenetworks/bfe/blob/develop/bfe_tls/handshake_messages.go#L27
Comment From: elindsey
JA3 fingerprinting in particular has proven to be quite useful, and we've been running a patched stdlib for a few years in order to support it.
I personally see less need to eg. more easily support SNI proxying - tlsrouter's support is already straightforward, and I'm not sure that's a common enough use case that it'd make sense to adjust the stdlib to better accommodate. Exposing the raw ClientHello would also bring up a discussion about how we want to handle the client random, if it should be included or sanitized.
For fingerprinting use cases, the only thing missing is exposing extensions. That's a small change to the API surface, in line with existing ClientHelloInfo fields, and addresses a concrete need. I'd be happy to contribute a patch if there was agreement in that direction.
cc @FiloSottile this seems to be in your wheelhouse. Do you have any thoughts on the request or how we can move this towards a decision?
Comment From: komuw
I ran into a need for this because I wanted to implement ja3 fingerprinting on my app.
Note that ja3 is now kind of going mainstream with AWS[1], cloudflare[2], fastly[3], among others, supporting it.
Also, it seems like caddy[4] ran into a need for something like this.
- https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-cloudfront-supports-ja3-fingerprint-headers/
- https://developers.cloudflare.com/bots/concepts/ja3-fingerprint/
- https://docs.rs/fastly/latest/fastly/struct.Request.html#method.get_tls_ja3_md5
- https://github.com/caddyserver/caddy/pull/1430
PS: @elindsey , is your patch somewhere public that someone like me can have a look?
Comment From: phuslu
As supplementary material, I also implement a high performance nginx module in https://github.com/phuslu/nginx-ssl-fingerprint
Comment From: mysticaltech
This is extremely useful in controlling bot abuses on a website!
Comment From: gopherbot
Change https://go.dev/cl/471396 mentions this issue: crypto/tls: expose extensions presented by client to GetCertificate
Comment From: komuw
if this is done, I think we should also update http.Request.TLS *tls.ConnectionState. This is because the initial ask in this issue was ClientHello can be useful for fingerprinting clients, as such I believe this is best utilised in the request-response lifecyle.
http.Request.TLS is already documented as
// allows HTTP servers and other software to record
// information about the TLS connection on which the request
// was received.
Comment From: bpowers
I think the problem with extending *tls.ConnectionState is that (a) it would either entail retaining the entire ClientHello, which can be up to 64k, or (b) require blessing a specific TLS fingerprinting technique. While the JA3 format is the most widely used today, its not the only format or dimension that one could imagine. With the change I posted above, folks who need fingerprinting can wrap http.Server such that their handler gets the fingerprint in each request's Context, while folks who don't need it pay no performance or memory penalty
Comment From: komuw
On Sat, 4 Mar 2023 at 20:22, Bobby Powers @.***> wrote:
I think the problem with extending *tls.ConnectionState is that (a) it would either entail retaining the entire ClientHello, which can be up to 64k, or (b) require blessing a specific TLS fingerprinting technique. While the JA3 format is the most widely used today, its not the only format or dimension that one could imagine.
I’m in agreement.
With the change I posted above, folks who need fingerprinting can wrap http.Server such that their handler gets the fingerprint in each request's Context, while folks who don't need it pay no performance or memory penalty
I think my imagination is failing me here. http.Server.Basecontext & http.Server.ConnContext are the ones that i can think of whose context will end up in the request. But those two methods do not have access to clientHello. Do you mind elaborating a little bit of what you have in mind? Thanks.
— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/32936#issuecomment-1454811580, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHMWUJ23QA645OMA726HVLW2N24HANCNFSM4H5WPGZQ . You are receiving this because you commented.Message ID: @.***>
Comment From: bpowers
@komuw I wrote an example, along with a test showing that its working: https://github.com/bpowers/go-fingerprint-example/blob/main/fingerprinting_server.go
The approach requires wrapping the underlying net.Conn, and then storing a shared struct on both the wrapped conn (Which is accessible from GetCertificate, which is called with the *tls.ClientHelloInfo) and the context returned from ConnContext, which is used to derive every request's context
Comment From: 1366613
Any progress on this?
I've developed a WAF for my own private use and just because the stdlib doesn't support having access to the raw TLS handshake, I had to go so deep intercepting all TLS handshakes with gopacket and correlating it with the HTTPS requests using some dirty tricks.
Comment From: phuslu
Finally I managed to extract clienthello raw bytes by MirrorHeaderConn trick.
func (ln TCPListener) Accept() (c net.Conn, err error) {
c, err = ln.Listener.Accept()
if err != nil {
return
}
if ln.MirrorHeader {
c = &MirrorHeaderConn{Conn: c, Header: nil}
}
if ln.TLSConfig != nil {
c = tls.Server(c, ln.TLSConfig)
}
return
}
type MirrorHeaderConn struct {
net.Conn
Header []byte
}
func (c *MirrorHeaderConn) Read(b []byte) (n int, err error) {
n, err = c.Conn.Read(b)
if c.Header == nil && n > 0 && err == nil {
c.Header = make([]byte, n)
copy(c.Header, b[:n])
}
return
}
For more details please see https://github.com/phuslu/liner/commit/594e552488d085165c634ab2f27cc1a670ff0d62 and https://github.com/phuslu/liner/commit/9f6ef2ddcc2137fbde2687403545d265250ae9ed
Comment From: gospider007
Packaging based on net. Conn, achieving ja3 fingerprint, http2 fingerprint, and ja4 fingerprint recognition https://github.com/gospider007/fp#quick-start-with-gin
Comment From: elindsey
@gospider007 your code appears to be making the assumption that the full ClientHello will be present in the first call to Read().
Comment From: gospider007
@gospider007 your code appears to be making the assumption that the full ClientHello will be present in the first call to Read().
Thank you for the reminder. It has now been fixed
Comment From: murph12F
is this been implemented yet? i can see here https://go-review.googlesource.com/c/go/+/471396 that it should have been added but dont seem to be in the documentation. I ve seen that somebody been using a patched stdlib, is that public? thx
Comment From: elindsey
The extensions change has not been merged, and it seems unlikely given the (very reasonable) upstream policy of not exposing API surface if it only supports finger printing use cases. The two options are to patch stdlib or to separately store and re-parse the client handshake bytes (dealing with the extra overhead that entails).
Comment From: murph12F
and can i patch the stdlib? would i have to just add the same changes of the change i ve sent before? sorry never patched stdlibs
Comment From: phuslu
@murph12F here're an example of "patched stdlib" golang https://github.com/phuslu/go
Comment From: joeshaw
You may be interested in the https://github.com/AGWA/tlshacks package, which adds net.Conn and net.Listener implementations that capture the ClientHello and parse it. It also contains functions for generating a JA3 fingerprint from it.
Comment From: 1366613
A fully working example of a pure Go method to process TLS client hello data:
package main
import (
"crypto/tls"
"encoding/json"
"github.com/projectdiscovery/sslcert"
"log"
"net/http"
"src.agwa.name/go-listener"
"src.agwa.name/tlshacks"
)
func handler(w http.ResponseWriter, req *http.Request) {
if req.URL.Path != "/" {
http.NotFound(w, req)
return
}
clientHello := req.Context().Value(tlshacks.ClientHelloKey).([]byte)
info := tlshacks.UnmarshalClientHello(clientHello)
w.Header().Set("Content-Type", "application/json")
encoder := json.NewEncoder(w)
encoder.SetIndent("", " ")
encoder.Encode(info)
}
func main() {
tlsOptions := sslcert.DefaultOptions
hostname := "localhost"
tlsOptions.Host = hostname
tlsConf, err := sslcert.NewTLSConfig(tlsOptions)
if err != nil {
log.Panic(err.Error())
}
httpServer := &http.Server{
Handler: http.HandlerFunc(handler),
ConnContext: tlshacks.ConnContext,
}
streamListener, err := listener.Open("tcp:443")
if err != nil {
log.Fatal(err)
}
defer streamListener.Close()
tlsListener := tls.NewListener(tlshacks.NewListener(streamListener), tlsConf)
log.Fatal(httpServer.Serve(tlsListener))
}
Comment From: rsc
This is an API change but looks like it was never officially proposed. Made this issue into a proposal. One external user asked about CL 471396 landing at some point.
Comment From: bobby-stripe
@rsc I think the original title of this issue and the proposed solution in CL 471396 (that I was the author of) have diverged slightly:
* originally the PR asked for the raw bytes of the ClientHello to be preserved and available on tls.ClientHelloInfo
* The CL "just" adds the list of extensions to tls.ClientHelloInfo in the order they were presented.
To me this felt lighter weight, doesn't force users to re-parse the ClientHello using some third party library, and is suitable for the JA3 TLS fingerprinting technique. There is a JA4 format - the major relevant change is that it requires access to the (first) ALPN value from the ClientHello. We have already parsed that by the time we're building ClientHelloInfo, it seems useful to add that to this proposal.
So the way I'd frame/word this proposal is:
* Add the slice of Extension IDs in the order presented by the ClientHello to ClientHelloInfo
* Add AlpnProtocols []string to ClientHelloInfo
And the concrete advantage this brings to Go users is the ability to fingerprint incoming TLS connections to servers, which is important for DDoS mitigation. If this isn't accepted, its possible for applications to implement fingerprinting themselves (example), but requires wrapping the TCP listener, buffering on the TCP connections, dual-parsing the ClientHello, and duplicating/getting the timeouts right when reading handshake packets. It would significantly reduce risk (especially in getting timeouts wrong/drift from Go defaults) + complexity for the *tls.ClientHelloInfo passed to the tls.Config.GetCertificate function to add the 2 additional fields needed for full fidelity fingerprinting.
Comment From: rsc
This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group
Comment From: rolandshoemaker
@bobby-stripe is the proposed AlpnProtocols different from the (unfortunately confusingly named) SupportedProtos?
Adding a slice of extension IDs seems reasonable, it'll require some piping of that information around to get it into the ClientHelloInfo, but that doesn't seem too cumbersome.
Comment From: bobby-stripe
@rolandshoemaker ah! I had grepped for ALPN and missed SupportedProtos -- thats exactly what we need I think! I have a draft PR that pipes the extension IDs through, its indeed not too cumbersome: https://go-review.googlesource.com/c/go/+/471396 (needs rebasing, I'll do that tonight). The small added test case currently looks to see if extensions is just non-empty; I suspect we want to be more explicit checking for a set of expected extensions, but you might have better or stronger opinions here
Comment From: rolandshoemaker
👍, will take a look. Updating the issue title to reflect the current state of this proposal: to add Extensions []uint16 to ClientHelloInfo.
Comment From: rsc
Have all remaining concerns about this proposal been addressed?
The proposal is to add one field “Extensions []uint16” to ClientHelloInfo and have the server populate it during the handshake.
Comment From: rsc
Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group
The proposal is to add one field “Extensions []uint16” to ClientHelloInfo and have the server populate it during the handshake.
Comment From: rsc
No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group
The proposal is to add one field “Extensions []uint16” to ClientHelloInfo and have the server populate it during the handshake.
Comment From: bobby-stripe
@rsc excited for this to be accepted! I've rebased CL 471396, but not sure how to properly tag someone like you or @rolandshoemaker to review it or kick off test execution
Comment From: bobby-stripe
@rolandshoemaker I've addressed your feedback on https://go-review.googlesource.com/c/go/+/471396 - I think its ready to submit (I need your help hitting the button, I don't have bits), but I'm also happy to make further revisions if you have additional comments or concerns! Thanks!
Comment From: rolandshoemaker
I've submitted it for you, thanks for the patch!