Pre-check

  • [x] I am sure that all the content I provide is in English.

Search before asking

  • [x] I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

jdk8，dubbo2.7.4.1，微服务集群部署，动态更新副本

Steps to reproduce this issue

日志：

[2025-05-06T08:04:57.476+08:00] [WARN] [ConsumeMessageThread_17-706] [com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler] [xxxx-yeying-xxxx-shadow-service-prodk8s-w7w5v] [xxxxxx] [xxxx-yeying-xxxx-shadow-service] [0a07457c1746489897462779148] [1.9] [] [TaskHandler.java,245,com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler,invokeExecutorAndUpdateTask] [任务失败] ## ' org.apache.dubbo.rpc.RpcException: No provider available in [invoker :interface com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade -> zookeeper://B注册中心/org.apache.dubbo.registry.RegistryService?application=xxxx-yeying-xxxx-shadow-service&check=false&dubbo=2.0.2&interface=com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&lazy=false&logger=log4j2&methods=queryProductPickItem,querySingleTableProductPickItem,queryPickedQty,insertProductPickItem&organization=xxxx&owner=xxxx-yeying-xxxx-shadow-service&pid=48&qos.enable=false®ister.ip=xxxx&release=2.7.4.1&side=consumer&sticky=false&timeout=3000×tamp=1746457332224,directory: org.apache.dubbo.registry.integration.RegistryDirectory@66d2885c, invoker :interface com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade -> zookeeper://B注册中心/org.apache.dubbo.registry.RegistryService?anyhost=true&application=xxxx-yeying-xxxx-shadow-service&bean.name=ServiceBean:com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&check=false&cluster=failfast&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=com.xxxx.yeying.xxxx.product.service.client.facade.ServiceStationPickItemFacade&lazy=false&loadbalance=random&logger=log4j2&methods=queryProductPickItem,querySingleTableProductPickItem,queryPickedQty,insertProductPickItem&organization=xxxx&owner=xxxx-yeying-xxxx-shadow-service&payload=60000000&pid=48&qos.enable=false®ister.ip=xxxx&release=2.7.4.1&remote.application=xxxx-yeying-product-xxxx-service&retries=0&rivers.tags=zxcgray&side=consumer&sticky=false&timeout=3000×tamp=1746457332224,directory: org.apache.dubbo.registry.integration.RegistryDirectory@42576db9] at org.apache.dubbo.rpc.cluster.support.RegistryAwareClusterInvoker.doInvoke(RegistryAwareClusterInvoker.java:59) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:248) at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:78) at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:55) at org.apache.dubbo.common.bytecode.proxy6.queryProductPickItem(proxy6.java) at com.xxxx.yeying.xxxx.shadow.service.integration.dubbo.ServiceStationPickClient.lambda$queryProductPickByRequestId$3b8e180$1(ServiceStationPickClient.java:89) at com.xxxx.zxc.xxxx.construction.utils.RpcClient.call(RpcClient.java:106) at com.xxxx.zxc.xxxx.construction.utils.RpcClient.call(RpcClient.java:96) at com.xxxx.zxc.xxxx.construction.utils.RpcClient.callForList(RpcClient.java:86) at com.xxxx.yeying.xxxx.shadow.service.integration.dubbo.ServiceStationPickClient.queryProductPickByRequestId(ServiceStationPickClient.java:89) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.queryPickOrder(StationSnowTaskExecutor.java:718) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.execute(StationSnowTaskExecutor.java:242) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor.execute(StationSnowTaskExecutor.java:100) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor$$FastClassBySpringCGLIB$$9b1bf0fc.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.datax.executor.StationSnowTaskExecutor$$EnhancerBySpringCGLIB$$61c7774b.execute() at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler.invokeExecutorAndUpdateTask(TaskHandler.java:215) at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler.submit(TaskHandler.java:95) at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler$$FastClassBySpringCGLIB$$b528b9d.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.datax.TaskHandler$$EnhancerBySpringCGLIB$$692c73ce.submit() at com.xxxx.yeying.xxxx.shadow.service.datax.TaskManager.submit(TaskManager.java:110) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer.onMessage(PickToFinishPickOrderConsumer.java:53) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer.onMessage(PickToFinishPickOrderConsumer.java:25) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer$$FastClassBySpringCGLIB$$6cdf58d2.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.xxxx.logistics.zxc.metric.MetricAdvice.handle(MetricAdvice.java:46) at sun.reflect.GeneratedMethodAccessor956.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at com.xxxx.yeying.xxxx.shadow.service.mq.PickToFinishPickOrderConsumer$$EnhancerBySpringCGLIB$$6eb79409.onMessage() at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer$DefaultMessageListenerConcurrently.consumeMessage(DefaultRocketMQListenerContainer.java:295) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

apollo： dubbo.registry.address=A注册中心 dubbo.provider.loadbalance=random dubbo.provider.retries=0 dubbo.registries.isDefault=true dubbo.registries.B.address=B注册中心

I have determined that there is no code or configuration that instructs the service to go to registry B to find a provider, and that in most cases it goes to registry A to find a provider, and only rarely to registry B. It is suspected that dubbo.registries.isDefault=true is interfering with some of the behaviors of dubbo to cause the bug that is causing dubbo to go to the wrong registry to find a provider.It is suspected that dubbo.registries.isDefault=true interferes with some behavior of dubbo and causes a bug, which causes dubbo to go to the wrong registry to find a provider.

来自AI的解读：

你观察到的现象——“大多数时候用对了 ZK，只有极少数情况用错了”——使得问题更加有趣和复杂。这确实强烈暗示了问题并非简单的静态配置错误，而是与某些动态条件、内部状态或潜在的 Bug 相关。 让我们深入思考为什么 dubbo.registries.isDefault=true 这个（可能不规范的）配置项会导致这种间歇性的错误行为： 非线程安全的初始化或状态变更: Dubbo 在启动时或运行时会根据配置初始化 Registry 和 RegistryDirectory 等组件。如果 dubbo.registries.isDefault=true 的处理逻辑存在并发问题（非线程安全），或者在处理 Apollo 配置动态更新时存在瑕疵，可能导致在某些竞争条件下，RegistryDirectory 错误地关联了 B 注册中心。 由于竞争条件发生的概率较低，这就解释了为什么大多数时候是正常的，只有偶尔会出错。错误日志中出现两个 RegistryDirectory 实例 (@54463380 和 @fabef2e) 可能就是这种内部状态混乱的体现。 缓存与刷新机制: Dubbo 内部有各种缓存（如注册中心地址列表、服务提供者列表等）。RegistryDirectory 会定期或在收到通知时刷新这些缓存。 如果 dubbo.registries.isDefault=true 干扰了注册中心的订阅逻辑或缓存刷新机制，可能导致在某个时间点，RegistryDirectory 的内部缓存包含了来自 B 的（无效）信息，并在那一刻的调用中被使用。当缓存下次正确刷新后，又恢复正常。 与 Apollo 配置动态推送的交互: Apollo 的配置是可以动态更新并推送给应用的。当 Dubbo 相关的配置发生变更（即使只是 dubbo.registries.isDefault=true 这一项存在），Dubbo 内部会触发相应的更新逻辑。 如果这个更新逻辑处理 dubbo.registries.isDefault=true 时存在 Bug，可能会短暂地导致内部注册中心状态不一致，使得某些线程在那个瞬间获取到了错误的注册中心引用。 集群容错或路由策略的副作用: 虽然我们没有看到明确的路由规则，但 Dubbo 的某些集群容错策略（如 RegistryAwareCluster）或负载均衡策略在处理多注册中心信息时，如果因为 dubbo.registries.isDefault=true 导致内部数据结构混乱，可能会在某些边界条件下（例如某个 ZK 连接短暂抖动、网络分区）触发 fallback 或探测逻辑，错误地尝试了 B。 RegistryDirectory 的生命周期管理: RegistryDirectory 实例通常与引用的服务接口绑定。它的创建、销毁、更新过程如果受到 dubbo.registries.isDefault=true 的干扰，可能导致在某些情况下（例如 Bean 的重新创建、作用域变化），获取到了一个状态错误的 RegistryDirectory 实例。 总结: dubbo.registries.isDefault=true 这个配置很可能是触发 Dubbo 内部某个潜在 Bug 或非预期行为的诱因。这个 Bug 可能与多注册中心处理、动态配置更新、并发控制或缓存刷新有关，导致在特定（可能与时间、并发、配置变更等相关的）条件下，消费者错误地尝试了非默认的 B 注册中心。

What you expected to happen

When there is no indication that an interface goes to a particular registry to find a provider, the default registry should be used instead of the unintended registry

Anything else

Occasionally, only 100 or so times in a day for hundreds of thousands of calls, so please tell me why!

Are you willing to submit a pull request to fix on your own?

  • [x] Yes I am willing to submit a pull request on my own!

Code of Conduct

Comment From: AlbumenJ

Check if the provider is publishing