Apache Dubbo [Bug] No provider available due to receiving go away frame immediately after successful connection

Pre-check

[x] I am sure that all the content I provide is in English.

Search before asking

[x] I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

Dubbo 3.3

Steps to reproduce this issue

it happened occationally, see details at https://github.com/apache/dubbo/actions/runs/13742828420/job/38434341835

What you expected to happen

this issue should not happen.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

[x] Yes I am willing to submit a pull request on my own!

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Comment From: zrlw

this issue happended again at https://github.com/apache/dubbo/actions/runs/14830129572/job/41629977785

2025-05-05T06:21:09.8275708Z [2/20] [dubbo-samples-triple:1/1] TEST FAILURE: Run tests failed, version: -Ddubbo.version=3.3.4-SNAPSHOT -Dcompiler.version=3.3.4-SNAPSHOT -Dspring.version=5.3.24 -Djava.version=21, please check logs: /home/runner/work/dubbo/dubbo/3-extensions/protocol/dubbo-samples-triple/target/logs

how to retrieve /home/runner/work/dubbo/dubbo/3-extensions/protocol/dubbo-samples-triple/target/logs? @Stellar1999

Comment From: Stellar1999

this issue happended again at https://github.com/apache/dubbo/actions/runs/14830129572/job/41629977785

2025-05-05T06:21:09.8275708Z [2/20] [dubbo-samples-triple:1/1] TEST FAILURE: Run tests failed, version: -Ddubbo.version=3.3.4-SNAPSHOT -Dcompiler.version=3.3.4-SNAPSHOT -Dspring.version=5.3.24 -Djava.version=21, please check logs: /home/runner/work/dubbo/dubbo/3-extensions/protocol/dubbo-samples-triple/target/logs

how to retrieve /home/runner/work/dubbo/dubbo/3-extensions/protocol/dubbo-samples-triple/target/logs? @Stellar1999

Need merge this PR @AlbumenJ https://github.com/apache/dubbo-samples/pull/1214

Comment From: Stellar1999

Damn, I haven't implemented this feature yet except for the PR-CI part — I'll add it a bit later.

Comment From: zrlw

i tested org.apache.dubbo.sample.tri.direct.TriDirectClientTest and found the issue will be triggered if TripleServerConnectionHandler throwed Http2Exception with Header size exceeded max allowed size (40960).

19:42:13.182 |-WARN  [NettyServerWorker-5-1] .transport.TripleServerConnectionHandler:    -|  [DUBBO] Channel:[id: 0x08bce973, L:/192.168.122.75:50052 - R:/192.168.122.75:57032] Error, dubbo version: 3.3.4-SNAPSHOT, current host: 192.168.122.75, error code: 4-14. This may be caused by , go to https://dubbo.apache.org/faq/4/14 to find instructions. 
io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (40960)
    at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:98) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:243) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.headerSizeExceeded(DefaultHttp2FrameReader.java:688) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.addFragment(DefaultHttp2FrameReader.java:717) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:473) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readContinuationFrame(DefaultHttp2FrameReader.java:602) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:271) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:39) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391) [netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451) [netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-base-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-base-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-base-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:359) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:167) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:383) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:577) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:552) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:493) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:470) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:204) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:175) [netty-transport-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073) [netty-common-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.2.0.Final.jar:4.2.0.Final]
    at java.lang.Thread.run(Thread.java:840) [?:?]
19:42:13.203 |-DEBUG [NettyServerWorker-5-1]                                H2_SERVER:214 -| [id: 0x08bce973, L:/192.168.122.75:50052 - R:/192.168.122.75:57032] OUTBOUND GO_AWAY: lastStreamId=0 errorCode=0 length=0 bytes=

the code of DefaultHttp2FrameReader line 271 is readContinuationFrame(in, listener):

private void processPayloadState(ChannelHandlerContext ctx, ByteBuf in, Http2FrameListener listener)
                    throws Http2Exception {
    ...
    switch (frameType) {
         ...
        case CONTINUATION:
                readContinuationFrame(in, listener);  <== line 271
                break;

Comment From: zrlw

The reason is the triple client sended header with large size meta data (131072) to server before calling HpackEncoder#setMaxHeaderListSize for the inboud settings (MAX_HEADER_LIST_SIZE=32768) received from the server, the header size exceeded max allowed size (40960), so the sever responded GO_AWAY frame.

set breakpoint at the client OUTBOUND:

09:33:19.333 |-DEBUG [NettyClientWorker-4-1]                                H2_CLIENT:214 -| [id: 0x760dcf34, L:/192.168.122.75:64974 - R:/192.168.122.75:50052] OUTBOUND HEADERS: streamId=3 headers=DefaultHttp2Headers[:scheme: http, :authority: 127.0.0.1:50052, :method: POST, :path: /org.apache.dubbo.sample.tri.Greeter/greet, content-type: application/grpc+proto, te: trailers, grpc-timeout: 300000m, tri-consumer-appname: org.apache.dubbo.sample.tri.direct.TriDirectClientTest, grpc-accept-encoding: gzip,snappy, timeout: 300000, large-size-meta: 
aaaaaaaaaa...] padding=0 endStream=false 
<== the length of 'aaaa...' is 131072


09:34:26.676 |-DEBUG [NettyClientWorker-4-1]                                H2_CLIENT:214 -| [id: 0x760dcf34, L:/192.168.122.75:64974 - R:/192.168.122.75:50052] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=8388608, MAX_FRAME_SIZE=8388608, MAX_HEADER_LIST_SIZE=32768}
<== received setting frame which contains MAX_HEADER_LIST_SIZE=32768

server log:

09:34:26.622 |-WARN  [NettyServerWorker-5-1] .transport.TripleServerConnectionHandler:    -|  [DUBBO] Channel:[id: 0x6b546e25, L:/192.168.122.75:50052 - R:/192.168.122.75:64889] Error, dubbo version: 3.3.4-SNAPSHOT, current host: 192.168.122.75, error code: 4-14. This may be caused by , go to https://dubbo.apache.org/faq/4/14 to find instructions. 
io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (40960)
    at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:98) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
    at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:243) ~[netty-codec-http2-4.2.0.Final.jar:4.2.0.Final]
<== received large header
... ... ...
09:34:26.670 |-DEBUG [NettyServerWorker-5-2]                                H2_SERVER:214 -| [id: 0x9e812d00, L:/192.168.122.75:50052 - R:/192.168.122.75:64974] OUTBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=8388608, MAX_FRAME_SIZE=8388608, MAX_HEADER_LIST_SIZE=32768}
<== sent setting frame to client

09:34:26.671 |-DEBUG [NettyServerWorker-5-2]                                H2_SERVER:214 -| [id: 0x9e812d00, L:/192.168.122.75:50052 - R:/192.168.122.75:64974] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=16646146
09:34:26.672 |-DEBUG [NettyServerWorker-5-1]                                H2_SERVER:214 -| [id: 0x6b546e25, L:/192.168.122.75:50052 - R:/192.168.122.75:64889] OUTBOUND GO_AWAY: lastStreamId=0 errorCode=0 length=0 bytes=
<== sent go away frame to client

Comment From: zrlw

Http2 client should wait for inbound SETTINGS frame before sending Headers frame, see details at https://www.rfc-editor.org/rfc/rfc7540#page-37

A SETTINGS frame MUST be sent by both endpoints at the start of a
   connection and MAY be sent at any other time by either endpoint over
   the lifetime of the connection.  Implementations MUST support all of
   the parameters defined by this specification.