Context I have a Spring Boot application running in AWS EKS with a Istio service mesh, calling a 3rd party Rest Api service using OkHttp version 4.12.0
Application details:
SpringBoot version 3.5.0
Java 21
Kotlin 1.9.23
istio-proxy: docker.io/istio/proxyv2:1.23.4
EKS version: 1.29
Issue After bumping my application from Spring Boot 3.3.4 to 3.5.0 (everything else being the same) the initial TCP connection blew out from ~500ms to 20 seconds
Spring Boot version: 3.3.4 SSL handshake completed in 437ms TCP connection completed in 519ms
Spring Boot version: 3.5.0 SSL handshake completed in 10017ms TCP connection completed in 20206ms
Steps taken so far Upon further investigation, I found that in Spring Boot 3.4, OkHttp Dependency management was removed.
Looking this the commit, it looks like the custom ssl configuration for the OkHttp Client was removed, probably explaining the slow ssl handshakes and tcp connection.
Even after adding a custom ssl config in my OkhttpClient(trying to copy what was removed in Spring Boot 3.4), didn't resolve the issue
@Bean
fun okHttpClient(): OkHttpClient {
val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
trustManagerFactory.init(null as KeyStore?)
val trustManager = trustManagerFactory.trustManagers.first() as X509TrustManager
val sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, arrayOf(trustManager), null)
return OkHttpClient.Builder()
.sslSocketFactory(sslContext.socketFactory, trustManager)
.addNetworkInterceptor(HttpLoggingInterceptor().setLevel(Level.BASIC))
.build()
}
I also tried swapping out to a apache http client org.apache.httpcomponents.client5:httpclient5:5.5, and the initial tcp connection time is still taking 20 seconds.
Comment From: wilkinsona
Thanks for the report, but I don't think there's enough information available here for us to diagnose the problem.
While you appear to have narrowed the cause down to the TCP connection and SSL handshake, we don't know anything about the SSL configuration. For example, the certificates that are involved. For us to be able to diagnose the problem, I think we'll need to be able to reproduce it. Without that, we can provide some guidance to allow you to self-diagnose.
You could try running with -Djavax.net.debug=all and comparing the output to identify where the two versions deviate. Profiling may also further narrow down the cause of the slowness. Another option would be to perform a binary search across the different versions of Spring Boot to identify exactly when the problem started to occur.
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: pexa-ashek
Hi @wilkinsona sorry for the late response to this. I have attached 2 ssl debug logs files comparing the 2 different versions of Spring Boot for the same business transactions (same EKS cluster with 'docker.io/istio/proxyv2:1.23.4' service mesh)
| Component | Spring Boot (3.3.4) | Spring Boot (3.5.0) | Performance Impact |
|---|---|---|---|
| Total Request Time | 2,665 ms | 23,457 ms | 8.8x slower |
| SSL Handshake | 447 ms | 10,026 ms | 22.4x slower |
| TCP Connection | 597 ms | 20,092 ms | 33.7x slower |
| DNS Lookup | 142 ms | 10 ms | 14.2x slower |
| ClientHello Size | 462 bytes | 797 bytes | 73% larger |
masked-ssl debug logs SpringBoot 3.3.4 bus trans thread.txt masked-ssl debug logs SpringBoot 3.5.0 bus trans thread.txt
Could the issue be Spring Boot 3.4+ SSL session attempting to work through Istio's service mesh, creating additional latency and complexity that wasn't present in the simpler 3.3.4 configuration.
The Spring Boot 3.4.3 Release notes alludes to this
Apache HTTP Components have changed defaults in the HttpClient relating to HTTP/1.1 TLS upgrades. Most proxy servers handle upgrades without issue, however, you may encounter issues with Envoy or Istio.
- Note: the builder.setProtocolUpgradeEnabled(false) spring config as suggested didn’t work