Spring Webflux /actuator/** endpoints can become unresponsive during high throughput, even when exposing a separate management port

During a high-throughput usage of one of our endpoints, we noticed that our metrics were failing to be scraped. The actual scenario then looks something like this:

After some deeper analysis, we found out that this is due to the fact that in Webflux, a high-throughput endpoint could potentially bottleneck the event loop and cause many other endpoints - including /actuator/**, to fail to be processed in time or at all.

Interestingly, the first indicator that this was happening was an AbortedException:

AbortedException: Connection has been closed BEFORE send operation

r.n.c.AbortedException: Connection has been closed BEFORE send operation
    at r.n.c.AbortedException.beforeSend(AbortedException.java:59)
    Suppressed: r.c.p.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ Handler Actuator web endpoint 'prometheus' [DispatcherHandler]
Original Stack Trace:
        at r.n.c.AbortedException.beforeSend(AbortedException.java:59)
        at r.n.h.s.HttpServerOperations.then(HttpServerOperations.java:685)
        at r.n.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:750)
        at r.n.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:739)
        at r.n.NettyOutbound.then(NettyOutbound.java:358)
        at r.n.h.s.HttpServerOperations.send(HttpServerOperations.java:544)
        at o.s.h.s.r.ReactorServerHttpResponse.writeWithInternal(ReactorServerHttpResponse.java:96)
        at o.s.h.s.r.AbstractServerHttpResponse.lambda$writeWith$2(AbstractServerHttpResponse.java:183)
        at o.s.h.s.r.AbstractServerHttpResponse.doCommit(AbstractServerHttpResponse.java:259)
        at o.s.h.s.r.AbstractServerHttpResponse.lambda$writeWith$5(AbstractServerHttpResponse.java:180)
        at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
        at r.c.p.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
        at r.c.p.Operators$ScalarSubscription.request(Operators.java:2570)
        at r.c.p.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
        at r.c.p.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
        at r.c.p.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
        at r.c.p.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
        at r.c.p.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
        at r.c.p.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101)
        at r.c.p.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152)
        at r.c.p.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
        at r.c.p.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101)
        at r.c.p.MonoJust.subscribe(MonoJust.java:55)
        at r.c.p.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
        at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
        at r.c.p.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
        at r.c.p.Operators$MonoInnerProducerBase.complete(Operators.java:2864)
        at r.c.p.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
        at r.c.p.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)
        at r.c.p.Operators$ScalarSubscription.request(Operators.java:2572)
        at r.c.p.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
        at r.c.p.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103)
        at r.c.p.Operators$MonoInnerProducerBase.request(Operators.java:2931)
        at r.c.p.Operators$MultiSubscriptionSubscriber.set(Operators.java:2366)
        at r.c.p.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2240)
        at r.c.p.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115)
        at r.c.p.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
        at r.c.p.FluxJust.subscribe(FluxJust.java:68)
        at r.c.p.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
        at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
        ... 14 frames truncated

which it seems was Prometheus trying to call the /actuator/prometheus endpoint, hitting some timeout, and cancelling the request.

What made this even harder to debug was the following log message from the org.springframework.web.server.adapter.HttpWebHandlerAdapter:

[4728cf52-74336] Error [org.springframework.http.converter.HttpMessageNotWritableException: No Encoder for [com.acme.error.handling.ErrorResponse] with preset Content-Type 'application/openmetrics-text;version=1.0.0;charset=utf-8'] for HTTP GET "/actuator/prometheus", but ServerHttpResponse already committed (200 OK)

(note: com.acme.error.handling.ErrorResponse is our own GlobalExceptionHandler trying to wrap the response error)

which is understandable given the above exception, although the logged response code (200 OK) seems a bit iffy.

While debugging this, we found out that we used the same server.port as well as management.server.port. According to the docs, setting an explicit management.server.port different from server.port should start a separate Netty which manages its own threadpool to protect against such use cases. Indeed, setting:

server.port: 8080
management.server.port: 8081

seems to do so:

2025-07-24T13:47:58.622Z  INFO 1 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2025-07-24T13:47:58.660326234Z 2025-07-24T13:47:58.660Z  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 14 endpoint(s) beneath base path '/actuator'
2025-07-24T13:47:58.682784995Z 2025-07-24T13:47:58.682Z  INFO 1 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8081

However, addressing this Netty:

does not solve the problem. In fact, reducing the standard Reactor Threadpool sizes, using e.g. JAVA_TOOL_OPTIONS=-Dreactor.schedulers.defaultPoolSize=4 -Dreactor.schedulers.defaultBoundedElasticSize=4 -Dreactor.schedulers.defaultBoundedElasticQueueSize=10

makes it relatively easy to reproduce an error like the following:

Test

2025-07-24T13:48:28.293Z ERROR 1 --- [or-http-epoll-3] a.w.r.e.AbstractErrorWebExceptionHandler : [c6475471-269]  500 Server Error for HTTP GET "/actuator/prometheus"
2025-07-24T13:48:28.310912574Z 
2025-07-24T13:48:28.310918808Z reactor.core.Exceptions$ReactorRejectedExecutionException: Task capacity of bounded elastic scheduler reached while scheduling 1 tasks (11/10)
2025-07-24T13:48:28.310923097Z  at reactor.core.Exceptions.failWithRejected(Exceptions.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310926836Z  Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
2025-07-24T13:48:28.310930136Z Error has been observed at the following site(s):
2025-07-24T13:48:28.310933355Z  *__checkpoint ⇢ Handler org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#handle(ServerWebExchange) [DispatcherHandler]
2025-07-24T13:48:28.310938710Z  *__checkpoint ⇢ org.springframework.web.filter.reactive.ServerHttpObservationFilter [DefaultWebFilterChain]
2025-07-24T13:48:28.310942459Z  *__checkpoint ⇢ HTTP GET "/actuator/prometheus" [ExceptionHandlingWebHandler]
2025-07-24T13:48:28.310946021Z Original Stack Trace:
2025-07-24T13:48:28.310949246Z      at reactor.core.Exceptions.failWithRejected(Exceptions.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310952578Z      at reactor.core.scheduler.BoundedElasticScheduler$BoundedScheduledExecutorService.ensureQueueCapacity(BoundedElasticScheduler.java:922) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310967825Z      at reactor.core.scheduler.BoundedElasticScheduler$BoundedScheduledExecutorService.submit(BoundedElasticScheduler.java:1022) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310971149Z      at io.micrometer.core.instrument.internal.TimedScheduledExecutorService.submit(TimedScheduledExecutorService.java:85) ~[micrometer-core-1.11.0.jar!/:1.11.0]
2025-07-24T13:48:28.310973763Z      at reactor.core.scheduler.DelegatingScheduledExecutorService.submit(DelegatingScheduledExecutorService.java:84) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310976424Z      at reactor.core.scheduler.Schedulers.directSchedule(Schedulers.java:1244) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310978942Z      at reactor.core.scheduler.BoundedElasticScheduler.schedule(BoundedElasticScheduler.java:304) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310981366Z      at reactor.core.scheduler.Schedulers$CachedScheduler.schedule(Schedulers.java:1163) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310984892Z      at reactor.core.publisher.MonoSubscribeOnCallable.subscribe(MonoSubscribeOnCallable.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310988045Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310990569Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310993206Z      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310995652Z      at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310998141Z      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311000808Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311003469Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311005829Z      at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311008374Z      at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311010813Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311013717Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311016306Z      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311018988Z      at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311025846Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311029219Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311032370Z      at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311034961Z      at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311037318Z      at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311039652Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311042683Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311045319Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311048008Z      at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311050758Z      at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:474) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311053379Z      at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311055908Z      at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311058402Z      at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311060865Z      at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:466) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311064703Z      at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311067310Z      at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311069869Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311072226Z      at reactor.core.publisher.MonoZip$ZipCoordinator.request(MonoZip.java:216) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311074567Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311077649Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311083407Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311086382Z      at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:125) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311088862Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311091365Z      at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311093732Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311096230Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311098592Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onComplete(MonoFlatMap.java:189) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311100958Z      at reactor.core.publisher.Operators.complete(Operators.java:137) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311103397Z      at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:121) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311105897Z      at reactor.core.publisher.Mono.subscribe(Mono.java:4485) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311108353Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311110738Z      at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311113103Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311115534Z      at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311117887Z      at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311120466Z      at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311122960Z      at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311125762Z      at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311128424Z      at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311130878Z      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311133899Z      at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311139036Z      at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311141672Z      at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311144040Z      at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311146438Z      at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2305) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311148971Z      at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:338) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311151631Z      at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311154081Z      at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311156523Z      at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2215) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311158896Z      at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311161282Z      at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311163833Z      at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311166542Z      at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311168922Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311171420Z      at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311173894Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311176263Z      at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311178550Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311180915Z      at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311183351Z      at reactor.core.publisher.Mono.subscribe(Mono.java:4485) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311185599Z      at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311190533Z      at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311193200Z      at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311195794Z      at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311198204Z      at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1006) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311200611Z      at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311203067Z      at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311205454Z      at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:621) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311207829Z      at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311210597Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311213117Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311215662Z      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311218313Z      at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:230) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311220637Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311223030Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311225555Z      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311227973Z      at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311230456Z      at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311232889Z      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311238413Z      at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311241410Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311243923Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311246769Z      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311249541Z      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311251964Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311254425Z      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311256933Z      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311259381Z      at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311267587Z      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311275056Z      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311278051Z      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311295568Z      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311301775Z      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311303333Z      at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]

where the Netty running on port 8081 suffers a RejectedExecutionException while the Netty running on port 8080 is getting a lot of requests.

I haven't looked much more deeply than this at the moment, but this seems to imply that despite starting a separate Netty Server, the same Reactor schedulers and threadpools are shared (possibly the shared Schedulers.parallel() instance) as opposed to using e.g. Schedulers.newParallel(), which IMO defeats the purpose of setting the management.server.port. Somewhat obviously, scraping /actuator/prometheus content via JMX as opposed to HTTP would be the only sensible workaround that I can see at the moment. This is a rather large change for something that seems like it should already work, however.

I have created a small demo project which showcases this issue. See the included README.md and docker-compose.yaml.

webflux-issue.zip

Comment From: filpano

Some further analysis (included in the demo):

Using the standard reactor threadpool (i.e. no subscribeOn(...) with a potentially blocking call (bad practice - just for testing purposes): -> Blocks /actuator/** calls
Using the standard reactor bounded elastic (using .subscribeOn(Schedulers.boundedElastic())): -> Blocks /actuator/** calls (somewhat surprisingly)
Using a custom bounded scheduler (using e.g. Schedulers.newBoundedElastic(4, 10, "customScheduler")): -> Does not block /actuator/** calls.

Just to clarify, there isn't anything special about the /actuator/** calls themselves - ordinary application endpoints would themselves also potentially be bullied out of a spot in the event loop by another, high-throughput endpoint.

The issue here, IMO, is that starting a separate Netty via management.server.port and scraping /actuator/** endpoints via that Netty does not seem to prevent this.