Crash report

Redis crashes during high-concurrency read/write benchmark when IO threads are enabled.

Full crash log (key section):

Additional information

  1. OS distribution and version
  2. Steps to reproduce (if any)

Client library

  • go-redis v9.11.0 (go1.23.10)

Environment

  • OS: Ubuntu 20.04 container on Kubernetes
  • Redis deployment: Helm chart / StatefulSet
  • Resources: 4C8G container, Redis standalone mode
  • CPU usage: spikes to ~6.8 cores during test

Steps to reproduce

  1. Deploy Redis 7.2.10 in K8s with the following config: ```conf io-threads 4 io-threads-do-reads yes

  2. Use a benchmark client (go-redis) with: 200 concurrent clients Random read/write ops (~100k ops)

  3. During test, Redis crashes and exits with refcount-related error. Expected

----- CURRENT CLIENT INFO ------ id=675 addr=245.0.0.144:40982 laddr=245.0.8.11:6379 fd=151 name= age=117 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=1036 multi-mem=0 rbs=1280 rbp=1033 obl=5 oll=0 omem=0 tot-mem=23716 events=r cmd=set user=default redir=-1 resp=3 lib-name=go-redis(,go1.23.10) lib-ver=9.11.0 argc: '3' argv[0]: '"set"' argv[1]: '"key:43408"' argv[2]: '"QOTCCIOBXYMUWXRYMSQCRVQXHLAIHYLEKIPZNKVOGUDIQNWBQCMGBIFECWNEMSQUUZKHHEHAMMDJARGDFSBMFKREEBJPUWHRFACGCXWEEXLABIWPZPUEDZKHLXSDPUHX"' 1:M 04 Aug 2025 11:37:04.343 # key 'key:43408' found in DB containing the following object: 1:M 04 Aug 2025 11:37:04.343 # Object type: 0 1:M 04 Aug 2025 11:37:04.343 # Object encoding: 0 1:M 04 Aug 2025 11:37:04.343 # Object refcount: 2

------ EXECUTING CLIENT INFO ------ id=675 addr=245.0.0.144:40982 laddr=245.0.8.11:6379 fd=151 name= age=117 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=1036 multi-mem=0 rbs=1280 rbp=1033 obl=5 oll=0 omem=0 tot-mem=23716 events=r cmd=set user=default redir=-1 resp=3 lib-name=go-redis(,go1.23.10) lib-ver=9.11.0 argc: '3' argv[0]: '"set"' argv[1]: '"key:43408"' argv[2]: '"QOTCCIOBXYMUWXRYMSQCRVQXHLAIHYLEKIPZNKVOGUDIQNWBQCMGBIFECWNEMSQUUZKHHEHAMMDJARGDFSBMFKREEBJPUWHRFACGCXWEEXLABIWPZPUEDZKHLXSDPUHX"' 1:M 04 Aug 2025 11:37:04.343 # key 'key:43408' found in DB containing the following object: 1:M 04 Aug 2025 11:37:04.343 # Object type: 0 1:M 04 Aug 2025 11:37:04.343 # Object encoding: 0 1:M 04 Aug 2025 11:37:04.343 # Object refcount: 2

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------ lazyfree-lazy-eviction no lazyfree-lazy-user-del no slave-read-only yes replica-read-only yes lazyfree-lazy-expire no lazyfree-lazy-server-del no client-query-buffer-limit 1gb activedefrag no io-threads-do-reads yes sanitize-dump-payload no proto-max-bulk-len 512mb io-threads 4 list-compress-depth 0 repl-diskless-load disabled repl-diskless-sync yes lazyfree-lazy-user-flush no

------ FAST MEMORY TEST ------ 1:M 04 Aug 2025 11:37:04.343 # Bio worker thread #0 terminated 1:M 04 Aug 2025 11:37:04.343 # Bio worker thread #1 terminated 1:M 04 Aug 2025 11:37:04.343 # Bio worker thread #2 terminated 1:M 04 Aug 2025 11:37:04.343 # IO thread(tid:140649982745400) terminated 1:M 04 Aug 2025 11:37:04.343 # IO thread(tid:140649961581368) terminated 1:M 04 Aug 2025 11:37:04.343 # IO thread(tid:140649961438008) terminated *** Preparing to test memory region 55c313106000 (2273280 bytes) *** Preparing to test memory region 55c313781000 (16384 bytes) *** Preparing to test memory region 7feabc200000 (1686110208 bytes) *** Preparing to test memory region 7feb20b00000 (1859125248 bytes) *** Preparing to test memory region 7feb8f900000 (256901120 bytes) *** Preparing to test memory region 7feb9ef27000 (135168 bytes) *** Preparing to test memory region 7feb9ef4a000 (135168 bytes) *** Preparing to test memory region 7feb9ef6d000 (135168 bytes) *** Preparing to test memory region 7feb9ef90000 (135168 bytes) *** Preparing to test memory region 7feb9fc00000 (8388608 bytes) *** Preparing to test memory region 7feba0425000 (421888 bytes) *** Preparing to test memory region 7feba048d000 (4096 bytes) *** Preparing to test memory region 7feba08d7000 (12288 bytes) *** Preparing to test memory region 7feba0a40000 (12288 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/redis/redis/issues

If a Redis module was involved, please open in the module's repo instead.

Suspect RAM error? Use redis-server --test-memory to verify it.

Some other issues could be detected by redis-server --check-system

Comment From: sundb

@yn-super did you use TLS?