Describe the bug
Hello! We have been observing a strange issue since upgrading our redis standalone servers from 6.2.5 to 7.2.5. Occasionally, one of our redis servers will start growing in RSS memory until the system is running out of memory. We have activedefrag enabled and it starts running during this time but it doesn't seem to help. Can someone help take a look and see what is going on? I captured some data when this occurred last. If we need more data I will have to wait until it happens again to get back with it.
MEMORY STATS
1) "peak.allocated"
2) (integer) 9155310640
3) "total.allocated"
4) (integer) 9088238672
5) "startup.allocated"
6) (integer) 859496
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 18185184
13) "cluster.links"
14) (integer) 0
15) "aof.buffer"
16) (integer) 0
17) "lua.caches"
18) (integer) 1416
19) "functions.caches"
20) (integer) 184
21) "db.0"
22) 1) "overhead.hashtable.main"
2) (integer) 547207768
3) "overhead.hashtable.expires"
4) (integer) 382011752
5) "overhead.hashtable.slot-to-keys"
6) (integer) 0
23) "overhead.total"
24) (integer) 948265800
25) "keys.count"
26) (integer) 10324751
27) "keys.bytes-per-key"
28) (integer) 880
29) "dataset.bytes"
30) (integer) 8139972872
31) "dataset.percentage"
32) "89.57447814941406"
33) "peak.percentage"
34) "99.26740264892578"
35) "allocator.allocated"
36) (integer) 9088373168
37) "allocator.active"
38) (integer) 12528521216
39) "allocator.resident"
40) (integer) 12667953152
41) "allocator-fragmentation.ratio"
42) "1.3785219192504883"
43) "allocator-fragmentation.bytes"
44) (integer) 3440148048
45) "allocator-rss.ratio"
46) "1.0111291408538818"
47) "allocator-rss.bytes"
48) (integer) 139431936
49) "rss-overhead.ratio"
50) "0.9998315572738647"
51) "rss-overhead.bytes"
52) (integer) -2134016
53) "fragmentation"
54) "1.393649697303772"
55) "fragmentation.bytes"
56) (integer) 3577581536
MALLOC-STATS
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g0"
Build-time option settings
config.cache_oblivious: false
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.opt_safety_checks: false
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: false
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 8
opt.percpu_arena: "disabled"
opt.oversize_threshold: 8388608
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Arenas: 9
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 9088438136, active: 12537638912, metadata: 139436448 (n_thp 0), resident: 12677103616, mapped: 12690853888, retained: 1994452992
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 74773, run_interval: 9706952082 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 15920251 21 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 8447672 11 0 0 0 0 3 0 0 0 0 0
ctl 31840495 43 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 30 59808 593498 1893267
muzzy: N/A 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 328396800 1215150 1 1214086 1 11320091 15 1215150 1 859012 1
total: 9088438136 555457892 765 447287792 616 67194024734 92575 86949748 119 30384046 41
active: 12537638912
mapped: 12690853888
retained: 1994452992
base: 139338144
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12677103616
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 15829375 21 0 0 0 0 2 0 0 0 0 0
extent_avail 17998331 24 0 0 2 0 104942 0 0 0 0 0
extents_dirty 22838844 31 0 0 58 0 119674 0 0 0 0 0
extents_muzzy 15864762 21 0 0 0 0 2 0 0 0 0 0
extents_retained 17626774 24 0 0 6 0 119698 0 0 0 0 0
decay_dirty 17250502 23 0 0 1 0 297186 0 0 0 0 0
decay_muzzy 15978550 22 0 0 0 0 296934 0 0 0 0 0
base 32529371 44 0 0 0 0 2 0 0 0 0 0
tcache_list 15829376 21 0 0 0 0 2 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 17489496 24 0 0 0 0 2 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 73137144 100 0 0 0 0 6578224 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 198481414 273 0 0 0 0 24699694 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 69605730 95 0 0 0 0 7114564 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 16602165 22 0 0 0 0 2 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 18185250 25 0 0 0 0 57184 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 94025222 129 0 0 0 0 10067686 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 37275666 51 0 0 0 0 2912754 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 26367533 36 0 0 0 0 1574130 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 18579568 25 0 0 0 0 370174 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 20741666 28 0 0 0 0 476220 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 32374751 44 0 0 0 0 1755476 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 88689395 122 0 0 0 0 12148526 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 52419929 72 0 0 0 0 6362682 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 60875629 83 0 0 0 0 1408028 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 17647301 24 0 0 0 0 228230 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 48252296 66 0 0 0 0 1153456 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 20171049 27 0 0 0 0 333398 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 17614587 24 0 0 0 0 63762 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 17760997 24 0 0 0 0 98412 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 17813467 24 0 0 0 0 51560 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 16834672 23 0 0 0 0 25354 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 16705338 23 0 0 0 0 14328 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 19406739 26 0 0 0 0 13934 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 15937093 21 0 0 0 0 7270 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 15906588 21 0 0 0 0 3350 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 15899656 21 0 0 0 0 2710 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 18909840 26 0 0 0 0 6118 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 15854369 21 0 0 0 0 1646 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 15892405 21 0 0 0 0 850 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 15863489 21 0 0 0 0 532 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 16896310 23 0 0 0 0 3288 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 15858668 21 0 0 0 0 400 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 15837168 21 0 0 0 0 238 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 15832719 21 0 0 0 0 110 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 15889896 21 0 0 0 0 1888 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 15833207 21 0 0 0 0 96 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 15869608 21 0 0 0 0 48 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 15832266 21 0 0 0 0 34 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[0]:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 30 59808 593486 1801107
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 59961344 1215136 1 1214074 1 11320077 15 1215136 1 859012 1
total: 8820002680 555457878 765 447287780 616 67194024720 92575 86949734 119 30384046 41
active: 12269203456
mapped: 12420321280
retained: 1677783040
base: 139244752
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12408573952
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7960125 10 0 0 0 0 1 0 0 0 0 0
extent_avail 10129062 13 0 0 2 0 104941 0 0 0 0 0
extents_dirty 14969568 20 0 0 58 0 119673 0 0 0 0 0
extents_muzzy 7995512 11 0 0 0 0 1 0 0 0 0 0
extents_retained 9757493 13 0 0 6 0 119697 0 0 0 0 0
decay_dirty 9234703 12 0 0 1 0 148985 0 0 0 0 0
decay_muzzy 8034898 11 0 0 0 0 148831 0 0 0 0 0
base 16790858 23 0 0 0 0 1 0 0 0 0 0
tcache_list 7960126 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 9620246 13 0 0 0 0 1 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 65267894 89 0 0 0 0 6578223 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 190612164 262 0 0 0 0 24699693 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 61736480 85 0 0 0 0 7114563 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 8732915 12 0 0 0 0 1 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 10316000 14 0 0 0 0 57183 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 86155972 118 0 0 0 0 10067685 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 29406416 40 0 0 0 0 2912753 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 18498283 25 0 0 0 0 1574129 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 10710318 14 0 0 0 0 370173 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 12872416 17 0 0 0 0 476219 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 24505501 33 0 0 0 0 1755475 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 80820145 111 0 0 0 0 12148525 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 44550679 61 0 0 0 0 6362681 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 53006379 73 0 0 0 0 1408027 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 9778051 13 0 0 0 0 228229 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 40383046 55 0 0 0 0 1153455 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 12301799 16 0 0 0 0 333397 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 9745337 13 0 0 0 0 63761 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 9891747 13 0 0 0 0 98411 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 9944217 13 0 0 0 0 51559 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 8965422 12 0 0 0 0 25353 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 8836088 12 0 0 0 0 14327 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 11537489 15 0 0 0 0 13933 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 8067843 11 0 0 0 0 7269 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 8037338 11 0 0 0 0 3349 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 8030406 11 0 0 0 0 2709 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 11040590 15 0 0 0 0 6117 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 7985119 11 0 0 0 0 1645 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 8023155 11 0 0 0 0 849 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 7994239 11 0 0 0 0 531 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 9027060 12 0 0 0 0 3287 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 7989418 11 0 0 0 0 399 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 7967918 10 0 0 0 0 237 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 7963469 10 0 0 0 0 109 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 8020646 11 0 0 0 0 1887 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 7963957 10 0 0 0 0 95 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 8000358 11 0 0 0 0 47 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 7963016 10 0 0 0 0 33 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[8]:
assigned threads: 0
uptime: 717554192152229
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 0 0 0 12 92160
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 0 0 0 0 0 0 0 0 0 0 0
large: 268435456 14 0 12 0 14 0 14 0 0 0
total: 268435456 14 0 12 0 14 0 14 0 0 0
active: 268435456
mapped: 270532608
retained: 316669952
base: 93392
internal: 0
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 268529664
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7869250 10 0 0 0 0 1 0 0 0 0 0
extent_avail 7869269 10 0 0 0 0 1 0 0 0 0 0
extents_dirty 7869276 10 0 0 0 0 1 0 0 0 0 0
extents_muzzy 7869250 10 0 0 0 0 1 0 0 0 0 0
extents_retained 7869281 10 0 0 0 0 1 0 0 0 0 0
decay_dirty 8015799 11 0 0 0 0 148201 0 0 0 0 0
decay_muzzy 7943652 11 0 0 0 0 148103 0 0 0 0 0
base 15738513 21 0 0 0 0 1 0 0 0 0 0
tcache_list 7869250 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
---
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
--- End jemalloc statistics ---
To reproduce
Unsure what causes this issue to reproduce.
Expected behavior
I expect active defrag running to help with external memory fragmentation
Comment From: RohanNagar
Happened again today, here's some more data. Running memory purge does not help either.
INFO ALL
# Server
redis_version:7.2.5
redis_git_sha1:ba8435db
redis_git_dirty:0
redis_build_id:3074ab5ecf741c27
redis_mode:standalone
os:Linux 4.18.0-553.22.1.el8_10.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.5.0
process_id:4376
process_supervised:systemd
run_id:e811f080646ab2775a9a7f8cace050c05a7dfc92
tcp_port:6379
server_time_usec:1729555801281191
uptime_in_seconds:1133405
uptime_in_days:13
hz:10
configured_hz:10
lru_clock:1502553
executable:/usr/sbin/redis-server
config_file:/etc/redis.conf
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=6379
listener1:name=unix,bind=/var/run/redis_sock/redis-6379.sock
listener2:name=tls,bind=0.0.0.0,port=7379
# Clients
connected_clients:737
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32768
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:9088235600
used_memory_human:8.46G
used_memory_rss:12318998528
used_memory_rss_human:11.47G
used_memory_peak:10576946736
used_memory_peak_human:9.85G
used_memory_peak_perc:85.92%
used_memory_overhead:492113104
used_memory_startup:859624
used_memory_dataset:8596122496
used_memory_dataset_perc:94.59%
allocator_allocated:9088345344
allocator_active:12174733312
allocator_resident:12318003200
total_system_memory:16524062720
total_system_memory_human:15.39G
used_memory_lua:45056
used_memory_vm_eval:45056
used_memory_lua_human:44.00K
used_memory_scripts_eval:1416
number_of_cached_scripts:3
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:77824
used_memory_vm_total_human:76.00K
used_memory_functions:184
used_memory_scripts:1600
used_memory_scripts_human:1.56K
maxmemory:9088234496
maxmemory_human:8.46G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.34
allocator_frag_bytes:3086387968
allocator_rss_ratio:1.01
allocator_rss_bytes:143269888
rss_overhead_ratio:1.00
rss_overhead_bytes:995328
mem_fragmentation_ratio:1.36
mem_fragmentation_bytes:3230781472
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:15918376
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:3
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:3855724342
rdb_bgsave_in_progress:0
rdb_last_save_time:1728422396
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:543749
total_commands_processed:14896539480
instantaneous_ops_per_sec:23278
total_net_input_bytes:1879293071321
total_net_output_bytes:507751839476
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:2897.23
instantaneous_output_kbps:775.56
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:334
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:147716
evicted_keys:186471833
evicted_clients:0
total_eviction_exceeded_time:17441
current_eviction_exceeded_time:0
keyspace_hits:3265529322
keyspace_misses:786302739
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:47437824
active_defrag_misses:644438217
active_defrag_key_hits:24995110
active_defrag_key_misses:40306569
total_active_defrag_time:682394696
current_active_defrag_time:60226072
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:7080869174
total_writes_processed:7080287466
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:5614803
reply_buffer_expands:5846381
eventloop_cycles:6389187531
eventloop_duration_sum:134357470753
eventloop_duration_cmd_sum:44913538693
instantaneous_eventloop_cycles_per_sec:8754
instantaneous_eventloop_duration_usec:31
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:c0aae4db6311168f7d6c653c893f1d239d1b4e76
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:70291.693001
used_cpu_user:66864.358432
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:70286.924357
used_cpu_user_main_thread:66862.782976
# Modules
# Commandstats
cmdstat_hget:calls=4014097266,usec=7164352079,usec_per_call=1.78,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=72511,usec=99811,usec_per_call=1.38,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=56674,usec=131705,usec_per_call=2.32,rejected_calls=0,failed_calls=0
cmdstat_del:calls=3002555646,usec=3529269530,usec_per_call=1.18,rejected_calls=0,failed_calls=0
cmdstat_acl|list:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_acl|save:calls=3,usec=5859,usec_per_call=1953.00,rejected_calls=0,failed_calls=0
cmdstat_acl|setuser:calls=3,usec=75,usec_per_call=25.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=18891,usec=4233845,usec_per_call=224.12,rejected_calls=0,failed_calls=0
cmdstat_set:calls=37734400,usec=50401772,usec_per_call=1.34,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=18891,usec=30860,usec_per_call=1.63,rejected_calls=0,failed_calls=0
cmdstat_latency|reset:calls=18891,usec=31469,usec_per_call=1.67,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=395,usec=2737807,usec_per_call=6931.16,rejected_calls=0,failed_calls=0
cmdstat_hdel:calls=200,usec=397,usec_per_call=1.99,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3284830546,usec=2776575965,usec_per_call=0.85,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=3,usec=139,usec_per_call=46.33,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=414564,usec=3532850,usec_per_call=8.52,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=504814497,usec=756635259,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_get:calls=37734400,usec=19816984,usec_per_call=0.53,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=74432,usec=8017,usec_per_call=0.11,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=4014097266,usec=41303237567,usec_per_call=10.29,rejected_calls=0,failed_calls=3
# Errorstats
errorstat_NOSCRIPT:count=3
# Latencystats
latency_percentiles_usec_hget:p50=2.007,p99=9.023,p99.9=17.023
latency_percentiles_usec_ping:p50=1.003,p99=6.015,p99.9=19.071
latency_percentiles_usec_auth:p50=2.007,p99=12.031,p99.9=28.031
latency_percentiles_usec_del:p50=1.003,p99=5.023,p99.9=16.063
latency_percentiles_usec_acl|list:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_acl|save:p50=1769.471,p99=2375.679,p99.9=2375.679
latency_percentiles_usec_acl|setuser:p50=9.023,p99=58.111,p99.9=58.111
latency_percentiles_usec_info:p50=208.895,p99=378.879,p99.9=679.935
latency_percentiles_usec_set:p50=1.003,p99=12.031,p99.9=24.063
latency_percentiles_usec_latency|latest:p50=1.003,p99=6.015,p99.9=17.023
latency_percentiles_usec_latency|reset:p50=1.003,p99=7.007,p99.9=18.047
latency_percentiles_usec_hgetall:p50=1.003,p99=43.007,p99.9=1002438.655
latency_percentiles_usec_hdel:p50=1.003,p99=9.023,p99.9=13.055
latency_percentiles_usec_expire:p50=1.003,p99=2.007,p99.9=14.015
latency_percentiles_usec_eval:p50=29.055,p99=82.431,p99.9=82.431
latency_percentiles_usec_hello:p50=7.007,p99=26.111,p99.9=40.191
latency_percentiles_usec_hset:p50=1.003,p99=10.047,p99.9=24.063
latency_percentiles_usec_get:p50=0.001,p99=2.007,p99.9=17.023
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_evalsha:p50=9.023,p99=35.071,p99.9=64.255
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=3232782,expires=3232782,avg_ttl=599116703
MEMORY STATS
1) "peak.allocated"
2) (integer) 10576946736
3) "total.allocated"
4) (integer) 9088233360
5) "startup.allocated"
6) (integer) 859624
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 15919144
13) "cluster.links"
14) (integer) 0
15) "aof.buffer"
16) (integer) 0
17) "lua.caches"
18) (integer) 1416
19) "functions.caches"
20) (integer) 184
21) "db.0"
22) 1) "overhead.hashtable.main"
2) (integer) 263527688
3) "overhead.hashtable.expires"
4) (integer) 211803704
5) "overhead.hashtable.slot-to-keys"
6) (integer) 0
23) "overhead.total"
24) (integer) 492111760
25) "keys.count"
26) (integer) 3232749
27) "keys.bytes-per-key"
28) (integer) 2811
29) "dataset.bytes"
30) (integer) 8596121600
31) "dataset.percentage"
32) "94.59413146972656"
33) "peak.percentage"
34) "85.92491912841797"
35) "allocator.allocated"
36) (integer) 9088323640
37) "allocator.active"
38) (integer) 12174626816
39) "allocator.resident"
40) (integer) 12317818880
41) "allocator-fragmentation.ratio"
42) "1.3395899534225464"
43) "allocator-fragmentation.bytes"
44) (integer) 3086303176
45) "allocator-rss.ratio"
46) "1.0117615461349487"
47) "allocator-rss.bytes"
48) (integer) 143192064
49) "rss-overhead.ratio"
50) "1.0001137256622314"
51) "rss-overhead.bytes"
52) (integer) 1400832
53) "fragmentation"
54) "1.355513095855713"
55) "fragmentation.bytes"
56) (integer) 3230986296
Comment From: sundb
@RohanNagar what was the memory footprint before upgration?
can you give the report from redis-cli --bigkeys?
Comment From: RohanNagar
Hi @sundb before the upgrade, memory would reach maxmemory and stay there. RSS memory would never climb like this, definitely never climbed much higher than the dataset memory.
Our keyspace is pretty large so redis-cli --bigkeys takes a long time. I will try to run it tomorrow and give the result. Are you thinking some large keys could cause this issue?
Comment From: sundb
feel like it doesn't relate to big keys. It may be related to the change in the size (224bytes and 320 bytes) of some structures, resulting in a higher fragmentation rate of these two bin. please wait for me to take some time to verify these two versions.
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 53006379 73 0 0 0 0 1408027 1 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 40383046 55 0 0 0 0 1153455 1 0 0 0 0
Comment From: RohanNagar
Hi @sundb, have you had a chance to take a look at this?
Comment From: sundb
@RohanNagar sorry, i'm busy in other place, please give me more time.
Comment From: reporter4u
I'm facing the same problem...
My Redis server worked well for some days but after fourteen days the problem appeared again:
# uptime
14:08:19 up 14 days, 5:50, 1 user, load average: 0.00, 0.00, 0.00
redis-server --version
Redis server v=6.2.18 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=43c2df230c37e8ca
127.0.0.1:6379> CONFIG GET activedefrag
1) "activedefrag"
2) "yes"
127.0.0.1:6379> CONFIG GET active-defrag-ignore-bytes
1) "active-defrag-ignore-bytes"
2) "104857600"
127.0.0.1:6379> info memory
# Memory
used_memory:6442054120
used_memory_human:6.00G
used_memory_rss:6777532416
used_memory_rss_human:6.31G
used_memory_peak:6454464936
used_memory_peak_human:6.01G
used_memory_peak_perc:99.81%
used_memory_overhead:32587504
used_memory_startup:812000
used_memory_dataset:6409466616
used_memory_dataset_perc:99.51%
allocator_allocated:6443080088
allocator_active:6933839872
allocator_resident:7006519296
total_system_memory:8025862144
total_system_memory_human:7.47G
used_memory_lua:30720
used_memory_lua_human:30.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:6442450944
maxmemory_human:6.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.08
allocator_frag_bytes:490759784
allocator_rss_ratio:1.01
allocator_rss_bytes:72679424
rss_overhead_ratio:0.97
rss_overhead_bytes:-228986880
mem_fragmentation_ratio:1.05
**mem_fragmentation_bytes:335519320**
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20496
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
Thanks for any help!
Comment From: sundb
@reporter4u this issue happened when upgrading to 7.2, but the version you're using is 6.2.
do you mean the defrag is ineffective?
i saw that the fragmention ratio is 8% from your input, which didn't look too high.
maybe you can share the INFO ALL and MEMORY malloc-stats
Comment From: reporter4u
@reporter4u this issue happened when upgrading to 7.2, but the version you're using is 6.2. do you mean the defrag is ineffective? i saw that the fragmention ratio is 8% from your input, which didn't look too high. maybe you can share the
INFO ALLandMEMORY malloc-stats
The defrag worked for nearly two weeks and now is ineffective. I haven't restarted the service: i have now 800 Mbyte of available RAM.
Maybe the defragmentation is not too high but is three time active-defrag-ignore-bytes which I espected to trig active defrag procedures. The problem is when I gave 6 gbyte of maxmemory and the rest to the SO and if Redis does not honor defragmentation I risk to go out of memory when i run other commands (for example python scripts running as Ansible host target).
# free
total used free shared buff/cache available
Mem: 7837756 6809952 423168 6480 604636 810160
Swap: 16777212 14848 16762364
These are the results o the commands:
124.0.0.1:6379> INFO ALL
# Server
redis_version:6.2.18
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:43c2df230c37e8ca
redis_mode:standalone
os:Linux 5.4.17-2136.344.4.1.el8uek.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.5.0
process_id:1052
process_supervised:systemd
run_id:0fe15f89734ca0a953ff225a2c928a85c6074f8f
tcp_port:6379
server_time_usec:1751027796302842
uptime_in_seconds:1232317
uptime_in_days:14
hz:10
configured_hz:10
lru_clock:6197332
executable:/usr/bin/redis-server
config_file:/etc/redis.conf
io_threads_active:0
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:40976
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:6441984384
used_memory_human:6.00G
used_memory_rss:6778433536
used_memory_rss_human:6.31G
used_memory_peak:6454464936
used_memory_peak_human:6.01G
used_memory_peak_perc:99.81%
used_memory_overhead:32625728
used_memory_startup:812000
used_memory_dataset:6409358656
used_memory_dataset_perc:99.51%
allocator_allocated:6443142664
allocator_active:6936264704
allocator_resident:7008944128
total_system_memory:8025862144
total_system_memory_human:7.47G
used_memory_lua:30720
used_memory_lua_human:30.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:6442450944
maxmemory_human:6.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.08
allocator_frag_bytes:493122040
allocator_rss_ratio:1.01
allocator_rss_bytes:72679424
rss_overhead_ratio:0.97
rss_overhead_bytes:-230510592
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:336451000
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:40992
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:42820421
rdb_bgsave_in_progress:0
rdb_last_save_time:1749795479
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:1202638
total_commands_processed:127211852
instantaneous_ops_per_sec:45
total_net_input_bytes:90373398960
total_net_output_bytes:1081949875940
instantaneous_input_kbps:4.50
instantaneous_output_kbps:1125.48
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:119230
evicted_keys:1979482
keyspace_hits:95739699
keyspace_misses:9809100
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:28751529
active_defrag_misses:9111180309
active_defrag_key_hits:8660483
active_defrag_key_misses:303746067
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:135565830
total_writes_processed:128530744
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:2180719d49cad4360b7436b2ba7400d7602167cf
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:3036.200037
used_cpu_user:5680.056479
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:3018.363009
used_cpu_user_main_thread:5675.180528
# Modules
# Commandstats
cmdstat_incr:calls=181,usec=600,usec_per_call=3.31,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=4280016,usec=7632663,usec_per_call=1.78,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=99264901,usec=899855907,usec_per_call=9.07,rejected_calls=0,failed_calls=0
cmdstat_slowlog:calls=2,usec=892,usec_per_call=446.00,rejected_calls=0,failed_calls=0
cmdstat_hmset:calls=4280016,usec=72473749,usec_per_call=16.93,rejected_calls=0,failed_calls=0
cmdstat_del:calls=11024,usec=122909,usec_per_call=11.15,rejected_calls=0,failed_calls=0
cmdstat_info:calls=24,usec=9697,usec_per_call=404.04,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=384,usec=1627,usec_per_call=4.24,rejected_calls=0,failed_calls=0
cmdstat_config:calls=26,usec=1395,usec_per_call=53.65,rejected_calls=0,failed_calls=0
cmdstat_set:calls=9050,usec=26391,usec_per_call=2.92,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=5390,usec=3175,usec_per_call=0.59,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=23,usec=19,usec_per_call=0.83,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=106,usec=235,usec_per_call=2.22,rejected_calls=0,failed_calls=0
cmdstat_get:calls=6276266,usec=16947480,usec_per_call=2.70,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=6539548,usec=252017362,usec_per_call=38.54,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=6539548,usec=3367873,usec_per_call=0.52,rejected_calls=0,failed_calls=0
cmdstat_mget:calls=834,usec=9649,usec_per_call=11.57,rejected_calls=0,failed_calls=0
cmdstat_command:calls=3,usec=5065,usec_per_call=1688.33,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=4509,usec=4505489,usec_per_call=999.22,rejected_calls=0,failed_calls=0
cmdstat_monitor:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=365386,expires=365362,avg_ttl=31275787546
127.0.0.1:6379> MEMORY malloc-stats
___ Begin jemalloc statistics ___
Version: "5.1.0-0-g0"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 8
opt.percpu_arena: "disabled"
opt.metadata_thp: "disabled"
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000)
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
Arenas: 8
Quantum size: 8
Page size: 65536
Maximum thread-cached size class: 229376
Number of bin size classes: 55
Number of thread-cache bin size classes: 55
Number of large size classes: 180
Allocated: 6443459960, active: 6936133632, metadata: 71668792 (n_thp 0), resident: 7007764480, mapped: 7011631104, retained: 4863492096
Background threads: 1, num_runs: 184514, run_interval: 6679229711 ns
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
background_thread 26930891 0 0 1 0 0 0
ctl 53861775 0 0 1 0 0 0
prof 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 1
uptime: 1232438213195905
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 0 175979 418577 1997875
muzzy: 10000 0 157709 358361 1543163
allocated nmalloc ndalloc nrequests
small: 5309949304 110747398 100151558 1403153747
large: 1133510656 1413192 1409303 1413192
total: 6443459960 112160590 101560861 1404566939
active: 6936133632
mapped: 7011631104
retained: 4863492096
base: 71611440
internal: 57352
metadata_thp: 0
tcache_bytes: 1268752
resident: 7007764480
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 13465445 0 0 1 0 0 0
extent_avail 14844385 0 0 234407 0 0 0
extents_dirty 18412111 1 17 351987 0 0 1
extents_muzzy 14622686 0 2 351207 0 0 0
extents_retained 14454779 3 4 311825 0 0 1
decay_dirty 14400287 0 2 369423 0 0 0
decay_muzzy 14302984 0 1 357361 0 0 0
base 26991021 0 0 3 0 0 0
tcache_list 13465446 0 0 1 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 25130896 23897258 20755896 58562025 3141362 386 8192 1 0.993 825748 212596 750 2512240 2756247594 0 0 24012853 0 0 0
16 1 28781760 23665284 21866424 267002082 1798860 441 4096 1 0.995 1731870 264989 1015 3822131 1560163612 0 0 15251407 0 0 0
24 2 86151024 31031232 27441606 237956553 3589626 441 8192 3 0.993 866973 271016 832 3143190 3136088740 0 0 25003473 0 0 0
32 3 1665984 4210957 4158895 143453896 52062 26 2048 1 0.977 2338532 182244 124 505570 54391461 0 0 6136011 0 0 0
40 4 624480 153096 137484 4925617 15612 2 8192 5 0.952 75806 81704 2 13970 28161019 0 0 5870745 0 0 0
48 5 1685472 616418 581304 487080361 35114 9 4096 3 0.952 326496 161964 27 97509 48326474 0 0 5996265 0 0 0
56 6 2673384 1585735 1537996 24612360 47739 6 8192 7 0.971 901441 171856 6 223019 61582896 0 0 5986701 0 0 0
64 7 24593856 3814460 3430181 39599864 384279 377 1024 1 0.995 745181 176425 1012 1168855 346035484 0 0 7993997 0 0 0
80 8 4930640 2344684 2283051 21038399 61633 16 4096 5 0.940 1073170 176463 39 408161 61173301 0 0 6164559 0 0 0
96 9 40889184 3725714 3299785 16753441 425929 209 2048 3 0.995 478637 174149 576 935161 372230460 0 0 8244243 0 0 0
112 10 5773600 792541 740991 5307312 51550 13 4096 7 0.968 271595 158024 44 199503 53980303 0 0 6187331 0 0 0
128 11 10226432 977840 897946 6710989 79894 159 512 1 0.981 194879 149899 550 359271 86645558 0 0 6529597 0 0 0
160 12 9878880 918142 856399 13185566 61743 31 2048 5 0.972 199239 154886 120 251727 78074851 0 0 6404195 0 0 0
192 13 13732416 800720 729197 8191298 71523 71 1024 3 0.983 170475 149326 314 223523 72103185 0 0 6326937 0 0 0
224 14 8845984 545799 506308 7134481 39491 20 2048 7 0.964 155711 132232 72 150833 49787336 0 0 6146019 0 0 0
256 15 7822336 419232 388676 3790739 30556 122 256 1 0.978 130795 117264 439 170066 43772848 0 0 6139317 0 0 0
320 16 24160640 680815 605313 8350497 75502 74 1024 5 0.996 205731 133886 203 218754 59482712 0 0 6257711 0 0 0
384 17 46997376 1431182 1308793 6930374 122389 242 512 3 0.987 261340 153710 804 401724 99875731 0 0 7059907 0 0 0
448 18 32565120 979506 906816 4644203 72690 75 1024 7 0.946 242535 152710 330 340839 78563431 0 0 6456017 0 0 0
512 19 28993536 1239385 1182757 4366970 56628 464 128 1 0.953 229917 157151 3262 386123 80497264 0 0 6702269 0 0 0
640 20 29203200 823818 778188 3417148 45630 90 512 5 0.990 222382 153468 445 278607 62187923 0 0 6340357 0 0 0
768 21 12021504 256198 240545 719834 15653 62 256 3 0.986 111098 106780 266 108164 27838346 0 0 5918685 0 0 0
896 22 7077504 218074 210175 540929 7899 30 512 7 0.514 112703 102723 97 74226 26283745 0 0 5670305 0 0 0
1024 23 4122624 105998 101972 304856 4026 63 64 1 0.998 75796 75791 304 31958 16290678 0 0 3425767 0 0 0
1280 24 20364800 331119 315209 650686 15910 63 256 5 0.986 108649 121270 482 118868 26398799 0 0 5934847 0 0 0
1536 25 45869568 395776 365913 840477 29863 237 128 3 0.984 120015 124471 960 172499 35486212 0 0 6125225 0 0 0
1792 26 72308992 539752 499401 1029437 40351 158 256 7 0.997 138558 137564 660 228710 45764737 0 0 6238545 0 0 0
2048 27 60315648 479251 449800 930450 29451 964 32 1 0.954 121804 133440 6091 187621 47240066 0 0 6139269 0 0 0
2560 28 34216960 148988 135622 243473 13366 107 128 5 0.975 66045 64302 345 73277 24901357 0 0 5805579 0 0 0
3072 29 48251904 222294 206587 467525 15707 472 64 3 0.519 91666 76120 981 94454 40434713 0 0 6029049 0 0 0
3584 30 29392384 97458 89257 169007 8201 65 128 7 0.985 53613 47316 245 41507 22926089 0 0 5642043 0 0 0
4096 31 15986688 65148 61245 83107 3903 260 16 1 0.938 34741 34727 1092 41073 18193131 0 0 4610167 0 0 0
5120 32 310978560 664271 603533 1576232 60738 953 64 5 0.995 164171 141336 3093 292059 58401736 0 0 6362759 0 0 0
6144 33 60739584 148772 138886 310982 9886 320 32 3 0.965 80191 72099 902 93768 23809581 0 0 5720603 0 0 0
7168 34 20063232 42482 39683 68142 2799 45 64 7 0.971 26316 24561 181 25052 15548617 0 0 2913331 0 0 0
8192 35 47349760 97604 91824 177283 5780 749 8 1 0.964 59347 59956 3227 59242 17701337 0 0 4406967 0 0 0
10240 36 47380480 112487 107860 203833 4627 255 32 5 0.567 64225 63210 749 60062 20646607 0 0 5170139 0 0 0
12288 37 34308096 56822 54030 85836 2792 192 16 3 0.908 32171 31521 840 36572 16726593 0 0 3845515 0 0 0
14336 38 24242176 34442 32751 56394 1691 55 32 7 0.960 19461 19529 252 19795 15141021 0 0 2471291 0 0 0
16384 39 25296896 27717 26173 47621 1544 411 4 1 0.939 19118 18918 1870 18394 14863121 0 0 2124761 0 0 0
20480 40 37621760 505175 503338 4378588 1837 121 16 5 0.948 321313 153699 6417 223571 15593645 0 0 2470717 0 0 0
24576 41 30081024 49862 48638 1080311 1224 159 8 3 0.962 37585 40367 1615 17885 14561323 0 0 1705735 0 0 0
28672 42 19353600 63731 63056 555544 675 46 16 7 0.917 55259 56579 942 14071 14148910 0 0 1013767 0 0 0
32768 43 19431424 68924 68331 377409 593 320 2 1 0.926 65001 65821 7257 8412 14018167 0 0 751895 0 0 0
40960 44 1224007680 374474 344591 9980955 29883 3738 8 5 0.999 150224 137112 16042 211251 27006344 0 0 5916211 0 0 0
49152 45 60899328 61310 60071 535399 1239 321 4 3 0.964 51123 52788 3147 18452 14194663 0 0 1103053 0 0 0
57344 46 484384768 206937 198490 575577 8447 1234 8 7 0.855 111064 107922 7008 123638 21124984 0 0 5419691 0 0 0
65536 47 429654016 191470 184914 359198 6556 6556 1 1 1 101098 109957 191470 0 20942293 0 0 5198569 0 0 0
81920 48 392151040 101153 96366 1692660 4787 1197 4 5 0.999 59881 65673 8610 57398 17360630 0 0 4142529 0 0 0
98304 49 103710720 78358 77303 288051 1055 574 2 3 0.918 70006 70267 8551 18446 14156107 0 0 953363 0 0 0
114688 50 58949632 88463 87949 250276 514 137 4 7 0.937 78192 78566 3724 19512 14286059 0 0 1150875 0 0 0
131072 51 66191360 57291 56786 127535 505 505 1 2 1 52697 53220 57291 0 14212407 0 0 960421 0 0 0
163840 52 18841600 58546 58431 336753 115 62 2 5 0.927 57207 57298 9086 2974 13685339 0 0 168987 0 0 0
196608 53 16318464 31162 31079 840232 83 83 1 3 1 29837 30044 31162 0 13639356 0 0 102885 0 0 0
229376 54 992739328 112071 107743 254980 4328 2205 2 7 0.981 81168 83325 14495 41431 16590478 0 0 3325727 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
262144 55 783810560 1109666 1106676 1109666 2990
327680 56 269352960 99905 99083 99905 822
393216 57 4718592 1400 1388 1400 12
458752 58 6422528 21550 21536 21550 14
524288 59 4194304 39986 39978 39986 8
655360 60 14417920 138168 138146 138168 22
786432 61 2359296 370 367 370 3
917504 62 0 629 629 629 0
1048576 63 0 38 38 38 0
1310720 64 10485760 492 484 492 8
1572864 65 0 52 52 52 0
1835008 66 0 3 3 3 0
2097152 67 0 2 2 2 0
2621440 68 0 41 41 41 0
3145728 69 0 1 1 1 0
3670016 70 29360128 780 772 780 8
4194304 71 8388608 3 1 3 2
5242880 72 0 106 106 106 0
---
--- End jemalloc statistics ---
Thanks.
Comment From: sundb
@reporter4u active-defrag-ignore-bytes does not mean that Redis will definitely defragment to such a low level. Instead, it indicates that defragmentation will only be executed if the fragmentation is higher than this.
from the malloc-stats, the vast majority of bins have been defragmented, and 8% fragmentation rate is actually not too high.
Comment From: reporter4u
@reporter4u
active-defrag-ignore-bytesdoes not mean that Redis will definitely defragment to such a low level. Instead, it indicates that defragmentation will only be executed if the fragmentation is higher than this. from the malloc-stats, the vast majority of bins have been defragmented, and 8% fragmentation rate is actually not too high.
Ok, but the memory is retained and who is doing this is Redis, despite INFO statistics. There are no other process who are using it.
The total of used memory does not match what Redis INFO is saying. 7.4 Gbyte (total ram) - 6 Gbyte (Redis maxmemory) - 200Mbyte (SO) = 1.2 Gbyte free
But now I have 800 Mbyte of available memory which is going down. If we add 300 Mbyte of fragmented ram we reach almost 1.2Gbyte that is ok.
If I restart Redis the count is ok for several days but when Redis stops to defragment the SO shows that there is few free memory.
At the moment the problem is not 800 Mbyte of free ram is that Redis does not start defragmentation process. And this will be a problem... It is not so good to do a manually service restart each time defragmentation stops to work because it is not predictable.
Comment From: sundb
@reporter4u what is the config active-defrag-threshold-lower? f you use the default
config, since your fragmentation rate is 8% which is smaller than default value(10%), defragmentation will not start at this time. You can reduce it to lower than 8,
Comment From: reporter4u
@reporter4u what is the config
active-defrag-threshold-lower? f you use the default config, since your fragmentation rate is 8% which is smaller than default value(10%), defragmentation will not start at this time. You can reduce it to lower than 8,
It seems what you are saying is that active-defrag-ignore-bytes and active-defrag-threshold-lower are in a sort of AND relationship. Is it correct? Because IMHO reading the config documentation this is not clear.
Thanks.
Comment From: sundb
# Minimum amount of fragmentation waste to start active defrag
# active-defrag-ignore-bytes 100mb
# Minimum percentage of fragmentation to start active defrag
# active-defrag-threshold-lower 10
The two of them have the "and" relationship