- I have a springboot application that acts as a websocket server using stomp. The embedded server is tomcat. I am using spring boot version 3.2.9 and springframework version 6.1.14.
- I also have a springboot client application that starts 5000 threads. Each thread opens a websocket connection with the server. The client is also using stomp.
-
My server machine has 62G ram. As websocket connections are being opened, the memory consumption in the server rises. At 5000 connections, I reach 20G memory usage (I configured my server app to run with -Xmx30g heap size to prevent server failure). When I allow less memory, the server throws OutOfMemoryError.
-
I saved a heap dump when reaching OOM error. Opened it with Eclipse MAT and saw that the dominant object is org.apache.tomcat.websocket.server.WsFrameServer.
-
So tomcat as a suspect, I changed my server application to use jetty instead of tomcat. Results: The same server application with jetty instead of tomcat uses a total of 1g ram for 5000 connected websocket clients, plus I made each client send a message to the server every 5 sec. The problem does not exist with jetty.
So is tomcat really a problem using stomp/ws, or is it the combination of my springboot version plus stomp?
Comment From: bclozel
It's hard to tell given the elements you have provided. Can you share more information on what's using memory? Maybe breaking down in more details how the memory is being used under that particular object.
It's not clear whether we're leaking memory somewhere, if we should improve our Tomcat support or if the problem is with Tomcat itself.
Thanks!
Comment From: tsiyona
When I analyze my heap dump in Eclipse Memory Analyzer I see the below info showing the dominator_tree. The dominator class is WsFrameServer. Please note the total Retained Heap value consumed by this class instances. This heap dump is from my websocket server running on tomcat. This heap dump profile was generated as a result of OOM Error.
- Does tomcat hold one instance of WsFrameServer per websocket connection made by the client?
- What tomcat configuration might I be missing that causes this class instances to inflate?
- Isn't 3M per instance too large? 5000 connections x 3M = 15.7GB which explains my OutOfMemoryError
Note that my websocket clients merely open a connection. They do not even explicitly send any messages to the server yet.
Class Name | Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------
org.apache.tomcat.util.threads.TaskThread @ 0x7fc20c7b8 http-nio-0.0.0.0-9084-exec-1 Thread| 120 | 3,184,464 | 0.02%
org.apache.tomcat.util.threads.TaskThread @ 0x7fc200810 http-nio-0.0.0.0-9084-exec-6 Thread| 120 | 3,184,304 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41484ae80 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x414f7cd88 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41556a250 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415577230 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4155787e0 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415a106c8 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415d3c4f8 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415e8f758 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416d08270 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416d14a70 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416f25978 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416f326c0 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41720a300 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4172179d8 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x417220690 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41722ce90 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4186047d0 | 120 | 3,154,904 | 0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x418610ff0 | 120 | 3,154,904 | 0.02%
Total: 20 of 232,891 entries; 232,871 more | 9,241,248 | 14,741,608,128 | 100.00%
-----------------------------------------------------------------------------------------------------------------------------------------