Spring Framework 6.2 introduces a buffer leak when using JettyClientHttpConnector with a Jetty's httpClient. This worked correctly in Spring Framework 6.1.
We tested this against all 6.2.x version and 6.1.x version. Always produce leaky buffers on 6.2.x. and it is always fine with versions 6.1.x.
In this example, we use ArrayByteBufferPool.Tracking to make it obvious there is a buffer leak, but in our actual project, we use the default implementation of ArrayByteBufferPool.
Here's a OneDrive link to an actual project but also the code here if you don't want to download external files:
# .\spring-memory-leak-test\src\main\java\com\test\spring\WebClientBuilderStoreExecutor.java
package com.test.spring;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.transport.HttpClientTransportDynamic;
import org.eclipse.jetty.io.ArrayByteBufferPool;
import org.eclipse.jetty.io.ArrayByteBufferPool.Tracking;
import org.eclipse.jetty.io.ClientConnector;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.springframework.http.client.reactive.JettyClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
public class WebClientBuilderStoreExecutor {
public static void main(String[] args) throws Exception {
ClientConnector clientConnector = new ClientConnector();
HttpClient httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));
Tracking tracking = new Tracking();
httpClient.setByteBufferPool(tracking);
WebClient client = WebClient.builder()
.clientConnector(new JettyClientHttpConnector(httpClient))
.build();
Map<Integer, String> results = new ConcurrentHashMap<>();
ExecutorService executor = Executors.newFixedThreadPool(50);
for (int i = 0; i < 10; i++) {
final int index = i;
executor.submit(
() -> {
String block =
client.get()
.uri("http://httpbin.org/anything")
.retrieve()
.bodyToMono(String.class)
.block();
results.put(index, block);
});
}
executor.shutdown();
executor.awaitTermination(60, TimeUnit.SECONDS);
System.out.println(
"Completed request. Results stored in HashMap with "
+ results.size()
+ " entries.");
System.out.println("Leaks count: " + tracking.getLeaks().size());
tracking.getLeaks().forEach(leaks -> System.out.println("Leaked buffers: " + leaks));
}
}
# \spring-memory-leak-test\pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test.spring</groupId>
<artifactId>spring-leak-test</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name></name>
<url>http://maven.apache.org</url>
<properties>
<jetty-reactive-httpclient.version>4.0.10</jetty-reactive-httpclient.version>
<!-- <spring-boot.version>3.5.4</spring-boot.version>
<spring.version>6.2.9</spring.version> -->
<spring-boot.version>3.3.12</spring-boot.version>
<spring.version>6.1.20</spring.version>
</properties>
<dependencies>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-reactive-httpclient</artifactId>
<version>${jetty-reactive-httpclient.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<version>${spring-boot.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>17</source>
<target>17</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.test.spring.WebClientBuilderStore</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.test.spring.WebClientBuilderStore</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Comment From: bclozel
Thanks for the detailed report and very useful and focused sample application, this makes our job easier. I can reproduce the issue. Most likely linked to #32097 as I can reproduce it with 6.2.0-M5 but not 6.2.0-M4. I'm looking into this.
Comment From: bclozel
Thanks again for the sample application @Kerry-G - I modified it a bit and pushed it to a public GitHub repository here: https://github.com/bclozel/framework-gh-35319
Dumping the actual leak information shows the following (for one buffer instance):
Buffer@243c0da3 of 16384 bytes on 2025-08-27T09:14:33.065679Z wrapping ReservedBuffer@146044d7[rc=1,DirectByteBuffer@59717824[p=606,l=606,c=16384,r=0]={HTTP/1.1 ...ing"\n}\n<<<>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00}]
DirectByteBuffer@59717824[p=606,l=606,c=16384,r=0]={HTTP/1.1 ...ing"\n}\n<<<>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00}
acquired at java.lang.Throwable: Acquired by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.<init>(ArrayByteBufferPool.java:878)
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking.acquire(ArrayByteBufferPool.java:845)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.newNetworkBuffer(HttpReceiverOverHTTP.java:202)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.acquireNetworkBuffer(HttpReceiverOverHTTP.java:179)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
4 retain(s)
java.lang.Throwable: Retained by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.retain(ArrayByteBufferPool.java:900)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:477)
at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1859)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1684)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:321)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.read(HttpReceiverOverHTTP.java:125)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:774)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Retained by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.retain(ArrayByteBufferPool.java:900)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onChunk(ResponseListeners.java:628)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer.onDemandCallback(ResponseListeners.java:502)
at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:177)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.invokeDemandCallback(HttpReceiver.java:811)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:792)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Retained by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.retain(ArrayByteBufferPool.java:900)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onChunk(ResponseListeners.java:628)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer.onDemandCallback(ResponseListeners.java:502)
at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:177)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.invokeDemandCallback(HttpReceiver.java:811)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:792)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Retained by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.retain(ArrayByteBufferPool.java:900)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.retain(ByteBufferChunk.java:160)
at org.springframework.core.io.buffer.JettyDataBuffer.<init>(JettyDataBuffer.java:58)
at org.springframework.core.io.buffer.JettyDataBufferFactory.wrap(JettyDataBufferFactory.java:95)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.emitOnNext(AbstractSinglePublisher.java:92)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.onNext(QueuedSinglePublisher.java:173)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.proceed(QueuedSinglePublisher.java:167)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.process(QueuedSinglePublisher.java:117)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.offer(QueuedSinglePublisher.java:41)
at org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor$ContentPublisher.read(ResponseListenerProcessor.java:227)
at org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor$ContentPublisher.run(ResponseListenerProcessor.java:195)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onDemandCallback(ResponseListeners.java:647)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onChunk(ResponseListeners.java:635)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer.onDemandCallback(ResponseListeners.java:502)
at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:177)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.invokeDemandCallback(HttpReceiver.java:811)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:792)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
4 release(s)
java.lang.Throwable: Released by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.release(ArrayByteBufferPool.java:915)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.release(ByteBufferChunk.java:166)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.release(ByteBufferChunk.java:166)
at org.eclipse.jetty.reactive.client.internal.ResponseEventPublisher$1.run(ResponseEventPublisher.java:75)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onDemandCallback(ResponseListeners.java:647)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer$ContentSource.onChunk(ResponseListeners.java:635)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer.onDemandCallback(ResponseListeners.java:502)
at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:177)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.invokeDemandCallback(HttpReceiver.java:811)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:792)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Released by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.release(ArrayByteBufferPool.java:915)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.release(ByteBufferChunk.java:166)
at org.eclipse.jetty.client.transport.ResponseListeners$ContentSourceDemultiplexer.onDemandCallback(ResponseListeners.java:504)
at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:177)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.invokeDemandCallback(HttpReceiver.java:811)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:792)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Released by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.release(ArrayByteBufferPool.java:915)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.reacquireNetworkBuffer(HttpReceiverOverHTTP.java:192)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:270)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.read(HttpReceiverOverHTTP.java:125)
at org.eclipse.jetty.client.transport.HttpReceiver$ContentSource.processDemand(HttpReceiver.java:774)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
java.lang.Throwable: Released by HttpClient@5d22bbb7-38
at org.eclipse.jetty.io.ArrayByteBufferPool$Tracking$Buffer.release(ArrayByteBufferPool.java:915)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.release(ByteBufferChunk.java:166)
at org.eclipse.jetty.io.internal.ByteBufferChunk$WithRetainable.release(ByteBufferChunk.java:166)
at org.springframework.core.io.buffer.JettyDataBuffer.release(JettyDataBuffer.java:107)
at org.springframework.core.io.buffer.DataBufferUtils.release(DataBufferUtils.java:610)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at org.springframework.core.io.buffer.DefaultDataBufferFactory.join(DefaultDataBufferFactory.java:122)
at org.springframework.core.io.buffer.JettyDataBufferFactory.join(JettyDataBufferFactory.java:100)
at org.springframework.core.io.buffer.JettyDataBufferFactory.join(JettyDataBufferFactory.java:31)
at org.springframework.core.io.buffer.DataBufferUtils.lambda$join$20(DataBufferUtils.java:685)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:283)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.emitOnComplete(AbstractSinglePublisher.java:104)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher$Complete.notify(QueuedSinglePublisher.java:188)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.proceed(QueuedSinglePublisher.java:163)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.process(QueuedSinglePublisher.java:117)
at org.eclipse.jetty.reactive.client.internal.QueuedSinglePublisher.complete(QueuedSinglePublisher.java:48)
at org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.onComplete(ResponseListenerProcessor.java:120)
at org.eclipse.jetty.client.transport.ResponseListeners.notifyComplete(ResponseListeners.java:353)
at org.eclipse.jetty.client.transport.ResponseListeners.lambda$addCompleteListener$7(ResponseListeners.java:338)
at org.eclipse.jetty.client.transport.ResponseListeners.notifyComplete(ResponseListeners.java:353)
at org.eclipse.jetty.client.transport.ResponseListeners.notifyComplete(ResponseListeners.java:345)
at org.eclipse.jetty.client.transport.HttpReceiver.terminateResponse(HttpReceiver.java:441)
at org.eclipse.jetty.client.transport.HttpReceiver.terminateResponse(HttpReceiver.java:423)
at org.eclipse.jetty.client.transport.HttpReceiver.lambda$responseSuccess$4(HttpReceiver.java:381)
at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:268)
at org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:168)
at org.eclipse.jetty.client.transport.HttpReceiver.responseHeaders(HttpReceiver.java:246)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:333)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:250)
at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
at java.base/java.lang.Thread.run(Thread.java:1447)
0 over-release(s)
So we are retaining the buffer when we are wrapping the original chunk JettyDataBufferFactory#wrap(Chunk) and later releasing it with DataBufferUtils.release. The problem is, according to the Jetty docs, a buffer needs to be released N+1 times. In the RetainableByteBuffer docs:
The
ByteBufferis released to aByteBufferPoolwhen#release()is called one more time than#retain(); in such case, the call to#release()returnstrue.
I think this was introduced in #32097, when an extra retain was added for wrapped buffers - see here vs here.
While an extra retain seems to be required on the server side, it is not on the client side.
I have modified JettyDataBufferTests to use a real retainable buffer instead of a mock and confirmed the behavior.
The fix should be available in SNAPSHOTs soon, to be released with our next maintenance release in September. Thanks again for your report!
Comment From: github-actions[bot]
Fixed via 764336f0f201c34ba0e636324e4b31da922c81c1
Comment From: Kerry-G
Thank you @bclozel for the solution and the clear explanation on the why that happened, I appreciate it!