On Redis 7 we first introduce the concept of Redis Functions. Redis function was built in a way that allows us to easily extends it with more languages support. One of the most desire language is JS. In this issue we will discuss some details about the JS engine and ask the community opinion about how to proceed.

We also consider to support JS on EVAL command, this will require a small refactoring to the EVAL code.

Choose JS Engine

When considering a JS engine, we look at the following criteria’s : * Binary size * Runtime memory consumption * Performance * Maintainability * Security * License

We perform some internal POC with 3 common JS engines: * Ducktape - Duktape * QuickJS - QuickJS Javascript Engine * V8 - V8 JavaScript engine

Unfortunately, neither of those engines were good enough. Lets extend the discussion of each of those engines:

V8

V8 is a JS engine that was written by google. It used by Chrome and NodeJS.

Advantages: * Performance - has a built in JIT * Maintainability - actively maintain by google and used all over * License - V8’s BSD-style license * Security - Widely used all over, considered a sandbox (though the recommendation is to run it on external processes to be 100% secure). For more information Untrusted code mitigations · V8 .

Disadvantages: * Binary size - 28MB after strip * Runtime memory consumption - Could get to 20 ~ 40 MB (It is possible to play with GC configuration and possibly shrink it).

Other notes to consider: * Supports WebAssembly * Written in C++ and require libstdc++

Ducktape

A widely used JS engine built to be embedded in C applications.

Advantages: * Binary size - 300K * Maintainability - actively maintain (though it seems like it is not keeping up with recent JS additions like async await and lambda functions) * Runtime memory consumption - Respectively small * License - MIT

Disadvantages: * Performance - Very slow, POC showed that it can get 20 times slower then Lua (the script execution part) and up to 3 times slower comparing end to end flow (from the moment client calls FCALL to the moment it gets the reply). * Security - Sandboxing is considered a work in progress Duktape Programmer's Guide

QuickJS

QuickJS is a small and embedded Javascript engine.

Advantages: * Binary size - 600K * Runtime memory consumption - Respectively small * License - MIT * Performance - As good as Lua

Disadvantages: * Maintainability - Not actively maintained and no commitment to maintain it * Security - Nothing is mentioned on the docs.

How to Proceed

We believe that V8 and Ducktape are not good options, V8 is to massively big to be embedded in Redis and Ducktape is to slow to be considered. We believe that the best option to proceed is analysing the QuickJS code and decide if it is considered a sandbox. Then, if it is, we can take ownership of the code base and maintain it as needed (Notice that bug fixes, like we already do today with Lua today, should be doable. But supporting new language features and follow the JS community will be much more challenging).

Other Options

We would like the community opinion before proceed.

Comment From: madolson

My initial thoughts: 1. I personally don't think we should add a second scripting language to Redis without a clear motivation, which isn't documented here. It adds more blast radius for CVEs, more support load, and more fragmentation in the community as people use different languages for scripts. 2. I ultimately don't see the syntax to be all that different from javascript compared to lua. A lot of the reason I think people like javascript has much less to do with the language and much more to do with the libraries and support infrastructure around it. I don't see a path for supporting that. I could be very wrong here, I've only use javascript for pet projects. LUA is also such a simple language. 3. The cross compilation to webassembly is interesting, and provides a nice security angle. I've looked into WASM as a redis engine in the past for performance, I think the WIT (WASM interface types) need to mature a bit and then it'll be a better fit. I think performance/security are important.

Comment From: oranagra

I think Lua does pose some barrier for people when they need to write scripts to redis (most are unfamiliar with the syntax), and JavaScript being so popular and also C-like syntax, would ease the pain on many. (maybe it'll make Redis the most loved database once again :wink:) maybe that alone is not a reason, but it's another factor.

Combining this with all the many requests to update Lua to 5.4 (which we can't do because it'll break compatibility with old scripts), i think adding a modern language and interpreter can help. here's one example: https://github.com/redis/redis/issues/5261

I think we can support many of the popular java script libraries, anything that's pure JS (people will bring it in on their own from NPM, no need to bundle it in redis), and i also think we can support development environments, by providing people with a way to test and debug their scripts locally, outside of redis.

Regarding CVEs, i think much of it depends on which interpreter we choose and how much effort we put into it's integration, it doesn't have to be anything like what we had with Lua (other than the plain fact that it's a script that might not be killable, but that's not related to the scripting engine)

Comment From: madolson

I think we can support many of the popular java script libraries, anything that's pure JS (people will bring it in on their own from NPM, no need to bundle it in redis), and i also think we can support development environments, by providing people with a way to test and debug their scripts locally, outside of redis.

I guess I don't know how npm works. Is there someway to extract everything out of all of your dependencies and generate a single JS blob and pass it into Redis? I assumed it was a package manager and everything was still kept in separate files, but maybe it's as simple as concatenating everything together?

Regarding CVEs, i think much of it depends on which interpreter we choose and how much effort we put into it's integration, it doesn't have to be anything like what we had with Lua (other than the plain fact that it's a script that might not be killable, but that's not related to the scripting engine)

Adding external dependencies adds risk, which I would always prefer to not take on in the project where possible. I think it's naive to think it will turn out much different than Lua.

Comment From: MeirShpilraien

I guess I don't know how npm works. Is there someway to extract everything out of all of your dependencies and generate a single JS blob and pass it into Redis? I assumed it was a package manager and everything was still kept in separate files, but maybe it's as simple as concatenating everything together?

There are a lot of bundlers that does just that, take a look at webpack for example.

Comment From: madolson

There are a lot of bundlers that does just that, take a look at webpack for example.

I will stand corrected and say that being able to easily use the large amount of libraries available for JS is a pretty compelling reason to add it to Redis.

Comment From: forkfork

It is rare to see large tooling exist around a Redis deployment - or when it does occur, it is not something that Redis actively encourages as a tool. Once a JS engine is deployed, I would expect Redis deployments in enterprises will tend to gain the large tooling / bundle sizes that we see in the Node.js world - which carries a large surface area of risk both wrt security & performance.

I personally would prefer that the surface area of supporting Redis setups (tooling, security considerations) be kept small.

Do we have any examples where this feature will be really needed?

Comment From: oranagra

@forkfork I think there are many community and company driven tools around redis (maybe not as much as other projects, but still), I'm not sure i'm aware of all of them (or actually i'm sure i'm not), but still i do hope for more. And i also think the redis project itself should include or promote some tools (see this as a planned improvement: #11122) One thing's for sure... if we don't add JS support to the core, there won't be any tools to support it 8-)

I'm not sure what you meant by enterprise tooling, but please note that in this case, we're talking of development phase tooling, not deployment tooling (which would remain a plain redis-cli, or other alternatives people use), IMO it doesn't look realistic or beneficial for a company to develop these and keep them close.

what surface area of risk do you refer to? can you give an example? was that related to the tooling or redis core? obviously any new feature adds surface area (LOLWAT and STRALGO included), we should weigh it in. in addition to the need to support the new feature, but i'm not sure this case doesn't justify that.

Regarding where it's needed, we are aware of people's frustration about Lua, as it poses an entry barrier for newcomers. Also the language is quite plain, and may luck some fancy features other more modern languages have (e.g. JS has async-await and other fancy features, which we might some day even integrate with redis). Lastly, our Lua VM is quite old and abandoned it doesn't get fixes any improvements anymore, i did give this example in my previous message.

Comment From: ShogunPanda

Since I was interested in implementing this, even just for fun, I have a simple question.

Can an engine be loaded as module or not?

In the first case, can you kindly point to the documentation that explains how? (I searched with no luck)

In the second case, do you have any plan to support it? I think this might be beneficial for end user to have custom implementation for specific use cases.

Thanks!

Comment From: oranagra

the module API is not (and not going to be) flexible enough to allow this to be done easily. of course you can implement a new set of commands can use RM_Call to do something similar, but not extend the existing scripting commands.

it is however possible that this can be something that's built as part of redis (uses redis internals), and can be optionally loaded at startup as a "module". similarly to what we did in #9320 see #10959

Comment From: piboye

i vote for quickjs

Comment From: Waldenesque

I'll try to keep this short. I'm an "outsider." I'm not a developer yet, though I have a long techie history with high exposure to dev topics, and I'm working toward some webdev skills. I see Redis as a key tech tool I may need to adopt in the future, however I haven't used it yet. In particular I noticed Redis Functions, and I'm curious about something I haven't yet seen discussed.

Why has njs from the nginx project not been mentioned?

Now I fully realize njs is an extremely specialized runtime, and nobody would use it for "general purpose" use cases targeted by V8 or Rhino or whatever. However, I get the impression that the way Lua scripting is typically used in Redis is relatively similar to the way njs is used in nginx. It seems close enough that it could be worth at least mentioning njs as a remotely possible candidate. Yet this hasn't happened, at least not that I noticed, so maybe there is something I don't understand. Perhaps the common use cases of Lua in Redis are wider than I realize, and njs is just too narrowly focused to be worth consideration. Or maybe Lua has better extensibility in terms of leveraging third-party packages, and njs has some implementation constraint that limits its ability to leverage JavaScript libraries. I have no idea, I'm just guessing blindly.

One thing I do happen to know is that njs code is apparently "friendly" enough to external adoption that AWS built CloudFront Functions on top of njs, though they don't admit that publicly. Another consideration that really stands out to me is that njs is maintained by the nginx project. People like Fabrice Bellard and Mike Pall are no doubt geniuses, but ask yourself why you're not running LuaJIT in Redis right now! Among other factors, lone geniuses are not necessarily the most reliable bets in terms of boring long-term maintenance. In this respect, it seems like njs could potentially be a safer bet.

I was hoping someone else who knows more than me would bring this up, but I haven't seen anything. I thought it was worth at least asking the question. Sorry if there is some obvious reason why this is a silly question. Thank you for your time. Also thanks for creating Redis Functions, whichever way the JS engine goes it will be a nice addition to this project.

Comment From: zuiderkwast

Any progress on this?

Just our perspecive: We're mainly interested in speed. Wring functions in something like WASM would be very nice, as long as it's really fast. Currently we require some custom modules just because Lua is too slow, which isn't very practical.

Comment From: mberndt123

I agree with @zuiderkwast that WebAssembly (WASM) should be considered as it has many advantages. - many available implementations, such as this fast, low-footprint one written in C - many languages targeting it - a vibrant and growing ecosystem around it - it's an open standard with broad industry support – it's not going anywhere

Adding support for any one programming language directly to Redis is something that I would advise against in this day and age. Most developers probably prefer writing their Redis functions in the same language that they use for their other day-to-day programming, and a universal language runtime like WASM is the tool to make that possible.

Comment From: saghul

QuickJS

QuickJS is a small and embedded Javascript engine.

Advantages:

* Binary size - 600K

* Runtime memory consumption - Respectively small

* License - MIT

* Performance - As good as Lua

Disadvantages:

* Maintainability - Not actively maintained and no commitment to maintain it

* Security - Nothing is mentioned on the docs.

👋 In case this is picked abck up. There is currently an actively maintained fork of QuickJS, with many projects now switching to it: https://github.com/quickjs-ng/quickjs/pulls

Disclaimer: I'm one of the fork maintainers.

Comment From: Akimotorakiyu

Any progress on this?

The homogeneity of front-end and back-end code at the business level is very convenient for full stack development. If Redis can support JS, it will be more efficient in complex business operations.

Comment From: aaronclawrence

My 2c: I have worked on a product that used Duktape first and then QuickJS. QuickJS was a massive upgrade in capability. I do believe that V8 would be an absolute behemoth and contrary to the general lightweight nature of Redis.