RESP is strongly FIFO, but occasionally - for a varied and complex set of reasons - protocol desync occurs, leading to incorrectly mating responses to their corresponding requests, in particular when using pipelining and/or multiplexing, or sometimes with redis proxy nodes between client and server. When this happens it might be noticed immediately, or it might not - depending on the commands and consequences. This can range from "mildly inconvenient" to "oh god, unplug everything right now and call the lawyers".
For people at the right-hand edge of that severity scale, I would like to propose, in some future version of redis, supporting entirely optional RESP3 attributes to communicate correlation, providing strong correctness guarantees - at the ticket price of "the client and server must support this, and you'll have some extra bytes in each message in both directions".
I'm also aware that attributes have silently disappeared from these docs, despite being here and here, and I know that no server or (AFAIK) clients currently handle them. Hence this isn't a "this week" thing. It is a "can we, as a community, work together to improve things in the long term, rather than constantly ignoring a problem because there is no immediate fix possible" thing.
Meat of the proposal:
Once inside RESP3 and having performed some version-check/handshake/whatever, clients can begin annotating messages with a preamble attribute, with a well-known key, cid perhaps ("correlation id"):
|1\r\n
+cid\r\n
$10\r\n
1029f737sd\r\n
{actual command here}
where the value passed (1029f737sd in the example) is opaque bytes with no meaning for the server. The server would, because of recognizing cid as a well-known meaning, echo the cid attribute as part of the response message:
|1\r\n
+cid\r\n
$10\r\n
1029f737sd\r\n
{actual response here}
With this, the client can have strong correctness guarantees about the response it is processing, and can immediately identify and burn a suspect connection.
Comment From: mgravell
Not sure this is really viable for a range of reasons; also discussing as above^^^ at the client-library level, but thought it was at least worth mentioning at the redis platform level - see if there's any interest or prior art.
Comment From: avislavkin
Hi @mgravell, could you plz share here more details about the use-case and environment where you experienced RESP "desync" issues? Thanks.
Comment From: mgravell
@avislavkin done
Comment From: mzimbres
RESP is strongly FIFO, but occasionally - for a varied and complex set of reasons - protocol desync occurs, leading to incorrectly mating responses to their corresponding requests, in particular when using pipelining and/or multiplexing, or sometimes with redis proxy nodes between client and server. When this happens it might be noticed immediately, or it might not - depending on the commands and consequences. This can range from "mildly inconvenient" to "oh god, unplug everything right now and call the lawyers".
@mgravell, Do you have any references about when and how desync occurs? I am the author of a Redis client that performs automatic pipelining and multiplexing and therefore I rely heavily on the correct arrival order.
Also, what other clients support multiplexing? I addition to Boost.Redis I remember only the NodeJs client.
Thanks.
@anarthal
Comment From: mgravell
The desync is currently believed to be a library bug in SE.Redis, something about memory buffers / pooling / GC compaction maybe. To the library impacted: critical - but to you: I think you can relax. However, in terms of "defence in depth": everything helps :)