Pre-check
- [x] I am sure that all the content I provide is in English.
Search before asking
- [x] I had searched in the issues and found no similar issues.
Apache Dubbo Component
Java SDK (apache/dubbo)
Dubbo Version
- Dubbo Java 3.2.10
- JDK jdk1.8.0_333
- Spring Boot 2.3.12.RELEASE
Steps to reproduce this issue
We application use Dubbo as RPC framework. Today, when dubbo.metrics publishEvent that a lot of threads have blocked on ConcurrentHashMap.computeIfAbsent().
The key thread call stack log.
"DubboServerHandler-192.168.106.148:8504-thread-499" Id=846 BLOCKED on java.util.concurrent.ConcurrentHashMap$Node@1001d0e owned by "DubboServerHandler-192.168.106.148:8504-thread-292" Id=624
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
- blocked on java.util.concurrent.ConcurrentHashMap$Node@1001d0e
at org.apache.dubbo.metrics.listener.AbstractMetricsListener.isSupport(AbstractMetricsListener.java:33)
at org.apache.dubbo.metrics.event.SimpleMetricsEventMulticaster.publishEvent(SimpleMetricsEventMulticaster.java:44)
at org.apache.dubbo.metrics.event.MetricsEventBus.lambda$before$7(MetricsEventBus.java:109)
at org.apache.dubbo.metrics.event.MetricsEventBus$$Lambda$2092/1902345087.run(Unknown Source)
at org.apache.dubbo.metrics.event.MetricsEventBus.tryInvoke(MetricsEventBus.java:96)
at org.apache.dubbo.metrics.event.MetricsEventBus.before(MetricsEventBus.java:109)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:80)
at org.apache.dubbo.metrics.filter.MetricsProviderFilter.invoke(MetricsProviderFilter.java:37)
The source code by the above key thread call stack log.
java.util.concurrent.ConcurrentHashMap#computeIfAbsent
org.apache.dubbo.metrics.listener.AbstractMetricsListener#isSupport
public abstract class AbstractMetricsListener<E extends MetricsEvent> implements MetricsListener<E> {
private final Map<Class<?>, Boolean> eventMatchCache = new ConcurrentHashMap<>();
/**
* Whether to support the general determination of event points depends on the event type
*/
public boolean isSupport(MetricsEvent event) {
// 事件类型匹配缓存
Boolean eventMatch = eventMatchCache.computeIfAbsent(
event.getClass(), clazz -> ReflectionUtils.match(getClass(), AbstractMetricsListener.class, event));
return event.isAvailable() && eventMatch;
}
@Override
public abstract void onEvent(E event);
}
org.apache.dubbo.metrics.event.SimpleMetricsEventMulticaster#publishEvent
public class SimpleMetricsEventMulticaster implements MetricsEventMulticaster {
@Override
public void publishEvent(MetricsEvent event) {
if (validateIfApplicationConfigExist(event)) return;
for (MetricsListener listener : listeners) {
// 指标监视器，是否支持这类事件
if (listener.isSupport(event)) {
listener.onEvent(event);
}
}
}
}
org.apache.dubbo.metrics.event.MetricsEventBus#before
public class MetricsEventBus {
/**
* Applicable to the scene where execution and return are separated,
* eventSaveRunner saves the event, so that the calculation rt is introverted
*/
public static void before(MetricsEvent event) {
MetricsDispatcher dispatcher = validate(event);
if (dispatcher == null) return;
// 通过异步线程发布事件
tryInvoke(() -> dispatcher.publishEvent(event));
}
}
org.apache.dubbo.metrics.filter.MetricsFilter#invoke(Invoker<?>, Invocation, boolean)
public class MetricsFilter implements ScopeModelAware {
public Result invoke(Invoker<?> invoker, Invocation invocation, boolean isProvider) throws RpcException {
if (rpcMetricsEnable) {
try {
// 请求相关的事件
RequestEvent requestEvent = RequestEvent.toRequestEvent(
applicationModel,
appName,
metricsDispatcher,
defaultMetricsCollector,
invocation,
isProvider ? PROVIDER : CONSUMER,
serviceLevel);
// 指标事件总线，发布事件
MetricsEventBus.before(requestEvent);
invocation.put(METRIC_FILTER_EVENT, requestEvent);
} catch (Throwable t) {
LOGGER.warn(INTERNAL_ERROR, "", "", "Error occurred when invoke.", t);
}
}
return invoker.invoke(invocation);
}
}
org.apache.dubbo.metrics.filter.MetricsProviderFilter#invoke
public class MetricsProviderFilter extends MetricsFilter implements Filter, BaseFilter.Listener {
@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
return super.invoke(invoker, invocation, true);
}
}
The following content is the full jstack log.
lefit-class.log
"DubboServerHandler-192.168.106.148:8504-thread-499" Id=846 BLOCKED on java.util.concurrent.ConcurrentHashMap$Node@1001d0e owned by "DubboServerHandler-192.168.106.148:8504-thread-292" Id=624
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
- blocked on java.util.concurrent.ConcurrentHashMap$Node@1001d0e
at org.apache.dubbo.metrics.listener.AbstractMetricsListener.isSupport(AbstractMetricsListener.java:33)
at org.apache.dubbo.metrics.event.SimpleMetricsEventMulticaster.publishEvent(SimpleMetricsEventMulticaster.java:44)
at org.apache.dubbo.metrics.event.MetricsEventBus.lambda$before$7(MetricsEventBus.java:109)
at org.apache.dubbo.metrics.event.MetricsEventBus$$Lambda$2092/1902345087.run(Unknown Source)
at org.apache.dubbo.metrics.event.MetricsEventBus.tryInvoke(MetricsEventBus.java:96)
at org.apache.dubbo.metrics.event.MetricsEventBus.before(MetricsEventBus.java:109)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:80)
at org.apache.dubbo.metrics.filter.MetricsProviderFilter.invoke(MetricsProviderFilter.java:37)
at com.leoao.starter.dubbo.generic.GenericFilterWrapper.invoke(GenericFilterWrapper.java:67)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ProfilerServerFilter.invoke(ProfilerServerFilter.java:66)
at com.leoao.starter.dubbo.generic.GenericFilterWrapper.invoke(GenericFilterWrapper.java:67)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:145)
at com.leoao.starter.dubbo.generic.GenericFilterWrapper.invoke(GenericFilterWrapper.java:67)
at com.leoao.lpaas.compatable.ContextFilterWrapper.invoke(ContextFilterWrapper.java:35)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:167)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:110)
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:205)
at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.dubbo.common.threadlocal.InternalRunnable.run(InternalRunnable.java:39)
at java.lang.Thread.run(Thread.java:750)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@3ab6beb1
What you expected to happen
We can find the root cause of the problem, the blocked behavior don't happen. Thx, bro!
Anything else
No response
Are you willing to submit a pull request to fix on your own?
- [x] Yes I am willing to submit a pull request on my own!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Comment From: songxiaosheng
Can you help me paste the complete stack? I need to find the thread holding the lock that has not been released yet,thanks
Comment From: bert82503
Can you help me paste the complete stack? I need to find the thread holding the lock that has not been released yet,thanks
I have upload the full stack, please find the lefit-class.log.
Comment From: zrlw
it might be issue#11986 of ConcurrentHashMap and we should use ConcurrentHashMapUtils to avoid such issues. e.g. https://github.com/apache/dubbo/issues/11986