The problem/use-case that the feature addresses
We're heavy users of Redis, with a large number of instances deployed in our production environment. In our daily work, we frequently encounter a common business need: combining INCR with EXPIRE.
For example, in a rate-limiting scenario, we need to limit the number of login attempts for players and set an expiration time for that count. This allows us to restrict a player's login attempts within a specific time frame. Previously, we accomplished this using SCRIPT LOAD and EVALSHA (because we need to incr count and set expiration atomically.)
This makes us wonder: Why doesn't Redis offer a dedicated INCREX command for this? It seems like this is a very common requirement across many different business use cases.
Description of the feature
A new command INCREX which incr a key and set expiration for it atomically.
command reference:
127.0.0.1:6379> increx nonexistent 100
(integer) 1
127.0.0.1:6379> ttl nonexistent
(integer) 99
127.0.0.1:6379> get nonexistent
"1"
127.0.0.1:6379> increx nonexistent 100
(integer) 2
127.0.0.1:6379> increx nonexistent 100
(integer) 3
127.0.0.1:6379> ttl nonexistent
(integer) 99
127.0.0.1:6379> get nonexistent
"3"
Alternatives you've considered
Currently we use script load + evalsha to do the same thing. If a dedicated INCREX command can be provided, it will be of great help in daily operation and maintenance.
Additional information
Comment From: raffertyyu
@sundb please have a review.
Comment From: sundb
@raffertyyu IMHO, If lua can support it without being complicated and without performance issues, I actually prefer not to add new commands to support it.
Comment From: raffertyyu
@raffertyyu IMHO, If lua can support it without being complicated and without performance issues, I actually prefer not to add new commands to support it.
# use increx
./memtier_benchmark --command='increx __key__ __int__' --test-time=300 --distinct-client-seed --randomize -R --data-size-range=10-100 --key-minimum=1 -c 20 -t 5 --pipeline=10 --key-prefix='kv_' --key-maximum=100000000
ALL STATS
==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
--------------------------------------------------------------------------------------------------
Increxs 359465.86 2.77937 2.55900 6.01500 9.47100 18919.65
Totals 359465.86 2.77937 2.55900 6.01500 9.47100 37839.29
# clear and restart redis-server
# use eval + script
./memtier_benchmark --command="eval \"redis.call('incr', KEYS[1]);redis.call('expire', KEYS[1], ARGV[1]);\" 1 __key__ __int__"
--test-time=300 --distinct-client-seed --randomize -R --data-size-range=10-100 --key-minimum=1 -c 20 -t 5 --pipeline=10 --key-prefix='kv_' --key-maximum=100000000 --select-db=0
ALL STATS
==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
--------------------------------------------------------------------------------------------------
Evals 152974.66 6.53318 5.85500 13.75900 17.53500 20002.61
Totals 152974.66 6.53318 5.85500 13.75900 17.53500 40005.22
# clear and restart redis-server
# use script load + evalsha
$ redis-cli
127.0.0.1:6379> script load "redis.call('incr', KEYS[1]);redis.call('expire', KEYS[1], ARGV[1]);"
"c824588fc7955ded9b8d39647d827ba103d944ba"
./memtier_benchmark --command='evalsha c824588fc7955ded9b8d39647d827ba103d944ba 1 __key__ __int__' --test-time=300 --distinct-client-seed --randomize -R --data-size-range=10-100 --key-minimum=1 -c 20 -t 5 --pipeline=10 --key-prefix='kv_' --key-maximum=100000000
ALL STATS
===================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
---------------------------------------------------------------------------------------------------
Evalshas 167194.14 5.97743 5.34300 12.79900 16.25500 17943.26
Totals 167194.14 5.97743 5.34300 12.79900 16.25500 35886.53
I conducted the above performance tests to verify the performance of different methods. Standalone commands clearly outperform Lua scripts in terms of OPS and latency.
Even ignoring the performance differences between standalone commands and Lua scripts for the same functionality, Lua scripts complicate business code logic, requiring more error handling to cope with the many possible error messages returned. For example, when Redis returns a NOSCRIPT error, we need to resend the script, which may introduce occasional latency. Standalone commands provide more explicit error messages, making them extremely useful for business logic, especially in web applications. Using Lua scripts is obviously a more expensive option, which brings more network bandwidth overhead, more complex error messages, and higher learning costs.
And to prove that this is a common need, I googled it and found these website: https://stackoverflow.com/questions/59606337/how-to-initiate-increment-and-set-expiration-in-the-same-redis-command https://groups.google.com/g/redis-db/c/jdfoC3aD8MA https://groups.google.com/g/redis-db/c/c9U7fo_F1Qs/m/GjtKgnv0AAAJ https://github.com/redis/go-redis/issues/1428 https://www.linkedin.com/pulse/handling-race-conditions-using-redis-atomic-ericson-cepeda-l%C3%B3pez https://github.com/redis/redis/issues/7631 https://github.com/redis/redis/pull/4870 https://github.com/redis/redis/issues/4423
It seems that this requirement has always been indirectly implemented through multi/exec or lua scripts. And the discuss under #4423 and other issues also seem to be very eager to add similar command or option.
So, please reconsider whether this command is acceptable.
Comment From: ptjm
I would think the better syntax would be to extend incr similarly to the way set was extended
INCR key [EX seconds|PX milliseconds|EXAT unix-time-seconds|PXAT unix-time-milliseconds|KEEPTTL] INCRBY key increment [EX seconds|PX milliseconds|EXAT unix-time-seconds|PXAT unix-time-milliseconds|KEEPTTL] INCRBYFLOAT key increment [EX seconds|PX milliseconds|EXAT unix-time-seconds|PXAT unix-time-milliseconds|KEEPTTL]
Comment From: LiorKogan
Thank you, @raffertyyu and @ptjm, for the suggestion. We agree this would be a valuable addition to Redis. Our initial plan is to support the following commands: DECR, DECRBY, INCR, INCRBY, and INCRBYFLOAT. We will review the preferred syntax options and follow up with an update here.
Comment From: raffertyyu
In my opinion, increx can meet almost all of these requirements. So generalizing this command is not that necessary. For example, many of our internal businesses require increx, but we've never had a request for similar incrpx/decrex commands, because the use cases for these commands are very specific and only apply to common scenarios like rate limiting.
Comment From: mgravell
If you're considering the optional argument approach, I'm just going to beat my drum on rate limiting. Just think how easy a basic rate limiter would be with
INCR key EX 60 LIMIT 100
Where LIMIT means "if the result would exceed this (in the relevant direction, accounting for sign in the incrby case), don't apply any change, including no change to TTL, and return something clear, maybe an -ERR"
(yes, I know there's more sophisticated rate limiting approaches, but sometimes simple is more than good enough)
Message ID: @.***>
Comment From: sundb
In my opinion, increx can meet almost all of these requirements. So generalizing this command is not that necessary. For example, many of our internal businesses require increx, but we've never had a request for similar incrpx/decrex commands, because the use cases for these commands are very specific and only apply to common scenarios like rate limiting.
some personal points: 1. If the command is extensible, we tend to extend it rather than add a new command. 2. We should not assume users' usage cases. If we only extend incr in the increx way, when we need to add new parameters in the future, we will no longer be able to expand it.