Pre-check
- [x] I am sure that all the content I provide is in English.
Search before asking
- [x] I had searched in the issues and found no similar issues.
Apache Dubbo Component
Java SDK (apache/dubbo)
Dubbo Version
Dubbo 3.2.12 + nacos-client 2.3.2, registration mode of the dubbo servers are interface，Jdk17
Steps to reproduce this issue
- start dubbo servers which register-mode are interface only.
- login nacos registation center, switch dubbo server interface status to offline.
- set dubbo client app breakpoints.
(1) at org.apache.dubbo.registry.integration.RegistryDirectory#refreshInvoker
private void refreshInvoker(List<URL> invokerUrls) {
Assert.notNull(invokerUrls, "invokerUrls should not be null");
if (invokerUrls.size() == 1
&& invokerUrls.get(0) != null
&& EMPTY_PROTOCOL.equals(invokerUrls.get(0).getProtocol())) {
// set first breakpoint at next refreshRouter(...) statement. <<<============================= SEE HERE !!!
refreshRouter(
BitList.emptyList(), () -> this.forbidden = true // Forbid to access
);
destroyAllInvokers(); // Close all invokers
(2) at org.apache.dubbo.registry.nacos.NacosRegistry#doSubscribe
private void doSubscribe(final URL url, final NacosAggregateListener listener, final Set<String> serviceNames) {
try {
if (isServiceNamesWithCompatibleMode(url)) {
/**
* Get all instances with serviceNames to avoid instance overwrite and but with empty instance mentioned
* in https://github.com/apache/dubbo/issues/5885 and https://github.com/apache/dubbo/issues/5899
*
* namingService.getAllInstances with
* {@link org.apache.dubbo.registry.support.AbstractRegistry#registryUrl}
* default {@link DEFAULT_GROUP}
*
* in https://github.com/apache/dubbo/issues/5978
*/
for (String serviceName : serviceNames) {
List<Instance> instances =
namingService.getAllInstances(serviceName, getUrl().getGroup(Constants.DEFAULT_GROUP));
notifySubscriber(url, serviceName, listener, instances);
}
// set second breakpoint at next for(...) statement. <<<============================= SEE HERE !!!
for (String serviceName : serviceNames) {
subscribeEventListener(serviceName, url, listener);
}
- start dubbo client app under debug mode, and it will run to first breakpoint which will set the forbidden flag to true.
- continue dubbo client app running to second breakpoint which will subscribe event listener.
- switch all dubbo server interface status to online at nacos registation center, and wait about a few seconds (let's say 10 seconds) to ensure nacos registration center finish notifying providers changing.
- continue dubbo client app running.
- as forbidden value of RegistryDirectory is always true, the dubbo client app will encouter the RpcException with 'No provider available from registry' whenever calling the dubbo server interface.
at org.apache.dubbo.registry.integration.DynamicDirectory#doList
@Override
public List<Invoker<T>> doList(
SingleRouterChain<T> singleRouterChain, BitList<Invoker<T>> invokers, Invocation invocation) {
// >>> the value of forbidden is always true. <<<
if (forbidden && shouldFailFast) {
// 1. No service provider 2. Service providers are disabled
throw new RpcException(
RpcException.FORBIDDEN_EXCEPTION,
"No provider available from registry " + this
+ " for service " + getConsumerUrl().getServiceKey() + " on consumer "
+ NetUtils.getLocalHost()
+ " use dubbo version " + Version.getVersion()
+ ", please check status of providers(disabled, not registered or in blocklist).");
}
What you expected to happen
moving subscribeEventListener calling statement of NacosRegistry#doSubscribe prior to notifySubscriber might solve this issue. just like ZookeeperRegistry#doSubscribe did.
Anything else
this issue has existed in dubbo2 and dubbo3 for a long time and many contributers proposed revision suggestions. I wonder why not made any revisions to this issue.
Are you willing to submit a pull request to fix on your own?
- [x] Yes I am willing to submit a pull request on my own!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Comment From: AlbumenJ
Actually, we don't know the revision from nacos server. Get before Sub or Sub before Get cannot solve the issue. Also, I found there maybe a concurrent issue there. According to nacos' maintainer, nacos will notify the latest data once Sub. So, for the result, even Get return empty provider, Sub will notify the latest providers and then Directory will be synced.
Comment From: zrlw
- the 'No provider‘ issue might be completely solved if nacos
notify the latest data once Sub. - since RegistryDirectory#notify is a synchronized method, why
there maybe a concurrent issue there?
Comment From: zrlw
- NacosRegistry#subscribeEventListener will create RegistryChildListenerImpl:
private void subscribeEventListener(String serviceName, final URL url, final NacosAggregateListener listener)
throws NacosException {
Map<NacosAggregateListener, Map<String, EventListener>> listeners =
nacosListeners.computeIfAbsent(url, k -> new ConcurrentHashMap<>());
Map<String, EventListener> eventListeners = listeners.computeIfAbsent(listener, k -> new ConcurrentHashMap<>());
// create RegistryChildListenerImpl
EventListener eventListener = eventListeners.computeIfAbsent(
serviceName, k -> new RegistryChildListenerImpl(serviceName, url, listener));
namingService.subscribe(serviceName, getUrl().getGroup(Constants.DEFAULT_GROUP), eventListener);
}
- RegistryChildListenerImpl will create a RegistryNotifier as it's notifier, if nacos notify the latest data once Sub, the notifier of RegistryChildListenerImpl will execute doNotify method when the notification is received:
public RegistryChildListenerImpl(String serviceName, URL consumerUrl, NacosAggregateListener listener) {
this.serviceName = serviceName;
this.consumerUrl = consumerUrl;
this.listener = listener;
this.notifier = new RegistryNotifier(getUrl(), NacosRegistry.this.getDelay()) {
@Override
protected void doNotify(Object rawAddresses) {
// call notifySubscriber here if nacos notify the latest data once Sub <<<=============== SEE HERE !!!
List<Instance> instances = (List<Instance>) rawAddresses;
NacosRegistry.this.notifySubscriber(consumerUrl, serviceName, listener, instances);
}
};
}
- Calling namingService.getAllInstances and notifySubscriber in NacosRegistry#doSubscribe will be redundant and might cause concurrent issue, i think such codes should be removed if nacos notify the latest data once Sub.
for (String serviceName : serviceNames) {
List<Instance> instances =
namingService.getAllInstances(serviceName, getUrl().getGroup(Constants.DEFAULT_GROUP));
notifySubscriber(url, serviceName, listener, instances);
}
Comment From: AlbumenJ
- the 'No provider‘ issue might be completely solved if nacos
notify the latest data once Sub.
Yes, but we should get the guarantee from name
- since RegistryDirectory#notify is a synchronized method, why
there maybe a concurrent issue there?
But why this issue exist? Because nacos do not notify the latest data once Sub?
Comment From: zrlw
since nacos do not notify the latest data once Sub, Sub before Get might solve this issue, at least alleviate the issue.
if Sub before Get at client starting,
1. if server status is offline yet, nacos will not notify when it get Sub request as the server status is not changed. but it will notify the client once the server status is changed to online no matter Get empty instances or not.
2. if server status is already online, Get after Sub will get server instances no matter nacos notification or not.
by the way, adding retry mechanism when setting forbidden to false might be another solution.
Comment From: zrlw
But why this issue exist? Because nacos do not notify the latest data once Sub?
Because nacos only notify one time for each server status changing event, and does not notify the clients which have NOT been subscribed. those clients which send Sub request after nacos notification will miss this server status changing notification forever.
Comment From: AlbumenJ
@KomachiSion PTAL
Comment From: KomachiSion
Sure, using getAllInstances with subscribe = true, the client will sub service by client without listener.
when call getAllInstances with subscribe = true and then call subscribe add listener first time, nacos client think caller has get the all instance and compare cached instances with subs new instances. If no changed, skip callback listener.
I think the way to solve problem is use getAllInstances with subscribe = false to get and then call subscribe .
Of cource, nacos-client also can do some enhancement to do callback sub at all time in new version by default.
Comment From: AlbumenJ
I think the way to solve problem is use
getAllInstanceswith subscribe = false to get and then callsubscribe.
@zrlw Would you like to patch it? You can submit to 3.2.x branch, and I will draft a release late this month.
Comment From: zrlw
i submitted #15126 for 3.2 and #15110 for 3.3,
but there are always sample test failures since 2 days ago and it seemed that dubbo-samples-configcenter-apollo sample test is always timeout.
Failed tests: 1
[3/34] [dubbo-samples-configcenter-apollo:1/1] TEST FAILURE: Run tests timeout, version: -Ddubbo.version=3.3.4-SNAPSHOT -Dspring.version=5.3.24 -Djava.version=8, please check logs: /home/runner/work/dubbo/dubbo/3-extensions/configcenter/dubbo-samples-configcenter-apollo/target/logs
```
Comment From: AlbumenJ
Fixed in https://github.com/apache/dubbo/pull/15126
Comment From: zrlw
----- Original Message ----- From: miracle @.> To: apache/dubbo @.> Cc: zrlw @.>, Mention @.> Subject: Re: [apache/dubbo] [Bug] some clients might always encounter the RpcException with 'No provider available from registry' if restarting all clients and servers which register-mode are interface (Issue #15107) Date: 2025-05-24 21:12 我们这个no provider问题和nacos注册中心实例缓存刷新机制以及我们的多机房部署网络环境有关，我自己也不能保证每次都能复现这个问题，此外no provider有很多种触发条件，你们遇到的问题未必和我们这个问题相同，建议1. 开debug日志开关，尝试从nacos和dubbo的日志信息分析问题原因；2. 尝试用最新版本的dubbo和nacos，看看问题是否得到解决。
Comment From: onceMisery
----- Original Message ----- From: miracle @.> To: apache/dubbo @.> Cc: zrlw @.>, Mention @.> Subject: Re: [apache/dubbo] [Bug] some clients might always encounter the RpcException with 'No provider available from registry' if restarting all clients and servers which register-mode are interface (Issue #15107) Date: 2025-05-24 21:12 我们这个no provider问题和nacos注册中心实例缓存刷新机制以及我们的多机房部署网络环境有关，我自己也不能保证每次都能复现这个问题，此外no provider有很多种触发条件，你们遇到的问题未必和我们这个问题相同，建议1. 开debug日志开关，尝试从nacos和dubbo的日志信息分析问题原因；2. 尝试用最新版本的dubbo和nacos，看看问题是否得到解决。
感谢回复！我遇到的现象很奇怪，当某个provider停机重启后，consumer均未重启。现象是有一台服务器上的所有consumer都会报相同的异常，而其他机器上的consumer都是正常的，关键是：我们并非部署在容器环境下，而是用传统的jar包部署在云上。