Spring Cloud Netflix RestTemplateEurekaHttpClient not closing HttpClient on shutdown

Spring Cloud 2021.0.0 Spring Boot 2.6.8

Bug Description: RestTemplateEurekaHttpClient is not closing HttpClient on shutdown. This leads to TCP CLOSE_WAIT connections to eureka server.

EurekaClient will shutdown when an exception occurs on an http request, but not shutdown HttpClient.

This bug is related to https://github.com/spring-cloud/spring-cloud-netflix/blob/27ac3379f6b391ccd605d212512f15c439825ecb/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateEurekaHttpClient.java#L203

In case of exception here: https://github.com/Netflix/eureka/blob/ed0da19ca1c049c87e3dbf75b6015c1861d5c2d0/eureka-client/src/main/java/com/netflix/discovery/shared/transport/decorator/RedirectingEurekaHttpClient.java#L96 new HttpClient will be created without closing the existing one - this causes CLOSE_WAIT connections

This supplier creates new CloseableHttpClient for every call to https://github.com/spring-cloud/spring-cloud-netflix/blob/27ac3379f6b391ccd605d212512f15c439825ecb/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateTransportClientFactory.java#L103

public class DefaultEurekaClientHttpRequestFactorySupplier implements EurekaClientHttpRequestFactorySupplier {

    @Override
    public ClientHttpRequestFactory get(SSLContext sslContext, @Nullable HostnameVerifier hostnameVerifier) {
        HttpClientBuilder httpClientBuilder = HttpClients.custom();
        if (sslContext != null) {
            httpClientBuilder = httpClientBuilder.setSSLContext(sslContext);
        }
        if (hostnameVerifier != null) {
            httpClientBuilder = httpClientBuilder.setSSLHostnameVerifier(hostnameVerifier);
        }
        CloseableHttpClient httpClient = httpClientBuilder.build();
        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
        requestFactory.setHttpClient(httpClient);
        return requestFactory;
    }

}

so in case of shutdown, currentEurekaClient shutdown don't closes connections: https://github.com/spring-cloud/spring-cloud-netflix/blob/27ac3379f6b391ccd605d212512f15c439825ecb/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateEurekaHttpClient.java#L203

possible solution will be trying to close the HttpClient: shutdown could be

public void shutdown() {
        Optional.of(unwrapRequestFactoryIfNecessary(restTemplate.getRequestFactory()))
                .filter(HttpComponentsClientHttpRequestFactory.class::isInstance)
                .map(HttpComponentsClientHttpRequestFactory.class::cast)
                .ifPresent(requestFactory-> {
                    try {
                        requestFactory.destroy();
                    } catch (Exception e) {
                    }
                });
    }

unwrapRequestFactoryIfNecessary https://github.com/spring-projects/spring-boot/blob/47516b50c39bd6ea924a1f6720ce6d4a71088651/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/web/client/RestTemplateBuilder.java#L746

spring-projects/spring-boot#31075

As you can see process open files increases over time until GC occurred

We can also create EurekaClientHttpRequestFactorySupplier which return the same ClientHttpRequestFactory, but it not a stable solution since we can't control eureka client code, maybe in case of exceptions connections not closes (hence not returns to PoolingHttpClientConnectionManager) - this can lead to no available connections in pool. In that case, restart is required

Original Issue: https://github.com/spring-cloud/spring-cloud-netflix/issues/4062

Comment From: BenEfrati

https://github.com/spring-projects/spring-boot/issues/31075#issuecomment-1128766582

the piece code instantiating the request factory should be in charge of closing resources properly

Comment From: OlgaMaciaszek

Hello, @BenEfrati , thanks for reporting it. Looks like a bug.

Comment From: sdzx3783

Recently, I encountered this problem in the production environment. I found that the HttpClient was created many times. The HttpClient connection pool did not enable the thread for clearing expired connections. The problem was solved by sharing one HttpClient and enabling the IdleConnectionEvaluator thread. The code is as follows:

@Configuration
public class EurekaConfig {

    private final AtomicReference<CloseableHttpClient> ref = new AtomicReference<>();

    @Bean
    public EurekaClientHttpRequestFactorySupplier defaultEurekaClientHttpRequestFactorySupplier() {
        return (sslContext, hostnameVerifier) -> {
            HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
            HttpClientBuilder httpClientBuilder = HttpClients
                    .custom()
                    .evictExpiredConnections()
                    .evictIdleConnections(30L, TimeUnit.SECONDS);
            if (sslContext != null) {
                httpClientBuilder = httpClientBuilder.setSSLContext(sslContext);
            }
            if (hostnameVerifier != null) {
                httpClientBuilder = httpClientBuilder.setSSLHostnameVerifier(hostnameVerifier);
            }
            if (ref.get() == null) {
                ref.compareAndSet(null, httpClientBuilder.build());
            }
            requestFactory.setHttpClient(ref.get());
            return requestFactory;
        };
    }
}
`

**Comment From: OlgaMaciaszek**

Hello @sdzx3783 - thanks for your comment. Would you like to submit a PR with a fix?

**Comment From: sdzx3783**

> 
I provide a temporary solution without changing the source code


**Comment From: dangkhoaphung**

Hi @OlgaMaciaszek , may I know which version will include the fix?

**Comment From: OlgaMaciaszek**

@dangkhoaphung  if the issue is added to an active backlog, it will be assigned a milestone. We're now prioritising  tasks for the upcoming release and will circle back to non-blocking issues later on. However, feel free to create a PR, and we will definitely review it.

**Comment From: OlgaMaciaszek**

Initial fix reverted due to https://github.com/spring-cloud/spring-cloud-netflix/issues/4275. Reopening.

**Comment From: impactCn**

Spring Cloud eureka client 4.1.3
Spring Boot 3.3.1

hi, I upgraded to 4.1.3 and still have a lot of close_wait problems.
@OlgaMaciaszek 

![Image](https://github.com/user-attachments/assets/4fb8d417-f0f6-4662-aafc-3731d3dc2f0d)

**Comment From: impactCn**

> Spring Cloud eureka client 4.1.3 Spring Boot 3.3.1
> 
> hi, I upgraded to 4.1.3 and still have a lot of close_wait problems. [@OlgaMaciaszek](https://github.com/OlgaMaciaszek)
> 
> ![Image](https://github.com/user-attachments/assets/4fb8d417-f0f6-4662-aafc-3731d3dc2f0d)

Is there any plan to fix this issue in which version?

**Comment From: OlgaMaciaszek**

Yes, @impactCn. the initial fix had to be reverted due to regression. We'll need to work on another fix, however it's not yet assigned to a new backlog. We some higher priority task right now, but I will try to look into it next month.


**Comment From: impactCn**

@OlgaMaciaszek I have a temporary solution to solve this problem, using webflux's webclient as the caller. I don't seem to have this problem in the test environment. I'll test it in the production environment tomorrow and take a look.



**Comment From: OlgaMaciaszek**

Sounds good @impactCn. Please take a look at that regression linked above as well to see if your solution is not doing the same. If it works fine, please feel free to create a PR or share the workaround.

**Comment From: impactCn**

eureka: client: webclient: enabled: true ``` @OlgaMaciaszek I used the above configuration and just observed it in the production environment for 30 minutes. There is no close_wait problem. And, I'll put it to tomorrow and see if there will be any problems with close_wait.

Comment From: OlgaMaciaszek

Ah, ok - that's a different thing - it's just using WebClient instead of HttpClient, but please verify if all your use-cases work, as the scope of integration (i.e.TransportFactories implementation) was not full for WebClient.

Comment From: ereborDeveloper

Hello! @OlgaMaciaszek do you have any updates on this one? Facing the same issue on netflix-eureka-client 3.1.1, spring boot 2.6.15. Which version should I upgrade to? Hope it will be possible for boot 2. Thank you!

Comment From: BenEfrati

Hello! @OlgaMaciaszek do you have any updates on this one? Facing the same issue on netflix-eureka-client 3.1.1, spring boot 2.6.15. Which version should I upgrade to? Hope it will be possible for boot 2. Thank you!

There is no fix, even for spring boot 3.x. You can supply your own EurekaClientHttpRequestFactorySupplier which can hold atomic reference to ClientHttpRequestFactory, so the RequestFactorySupplier will return the same instance for every call, rather than create new http client which will never be closed.

https://github.com/spring-cloud/spring-cloud-netflix/issues/4103#issuecomment-1294394463

Comment From: r-prates

Recently, I encountered this problem in the production environment. I found that the HttpClient was created many times. The HttpClient connection pool did not enable the thread for clearing expired connections. The problem was solved by sharing one HttpClient and enabling the IdleConnectionEvaluator thread. The code is as follows:

``` @Configuration public class EurekaConfig {

private final AtomicReference<CloseableHttpClient> ref = new AtomicReference<>();

@Bean
public EurekaClientHttpRequestFactorySupplier defaultEurekaClientHttpRequestFactorySupplier() {
    return (sslContext, hostnameVerifier) -> {
        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
        HttpClientBuilder httpClientBuilder = HttpClients
                .custom()
                .evictExpiredConnections()
                .evictIdleConnections(30L, TimeUnit.SECONDS);
        if (sslContext != null) {
            httpClientBuilder = httpClientBuilder.setSSLContext(sslContext);
        }
        if (hostnameVerifier != null) {
            httpClientBuilder = httpClientBuilder.setSSLHostnameVerifier(hostnameVerifier);
        }
        if (ref.get() == null) {
            ref.compareAndSet(null, httpClientBuilder.build());
        }
        requestFactory.setHttpClient(ref.get());
        return requestFactory;
    };
}

} ` ```

Hello! @sdzx3783, can you please help me on understanding where should I add this configuration? Should it be on the eureka server side? Or on every eureka client that connects to the eureka server? Thank you