June 16th, 2022 @ justine's web page

redbean 2.0

redbean is a webserver in a zip executable that runs on six operating systems. The basic idea is if you want to build a web app that runs anywhere, then you download the redbean.com file, put your .html and .lua files inside it using the zip command, and then you've got a hermetic app you can deploy and share. I introduced this web server about a year ago on Hacker News, where it became the third most upvoted hobby project of all time.

Over the last year, we've turned redbean into more than a hobby project. It's grown to become a 1.9mb file that self-hosts a Lua + SQLite development stack. There's builtin MbedTLS support. It does sandboxing. It has argon2 password hashing. It can geolocate IPs with MaxMind. It has a readline-like REPL. You can use it as a Lua shebang interpreter. It has an easy-mode API and a Fullmoon web framework for high-level development. It also has a hard-mode API that provides direct access to Cosmopolitan Libc Unix system calls. You can use Unix on Windows, or even from JavaScript too, since redbean is great for spinning up a web GUI in Chrome via localhost. You can also use redbean as a production web server on the public-facing internet. I stand by that statement since I eat my own dogfood. redbean hosts all my websites now, including this one (justine.lol). There's no need for a proxy like nginx; redbean is vertically integrated.

Download   [Linux] [Windows] [MacOS] [FreeBSD] [OpenBSD] [NetBSD]

Your redbean supports x86-64 Linux, MacOS, FreeBSD, NetBSD, or OpenBSD. Visit redbean.dev/2.0.html to download the release binary. redbean is permissively licensed under the ISC license. The source code is available on GitHub. Instructions for building redbean from source on Linux are available at redbean.dev. 

curl https://redbean.dev/redbean-demo-2.0.19.com >redbean.com
chmod +x redbean.com
./redbean.com -v

PowerShell users can use: 

wget -O redbean.com https://redbean.dev/redbean-demo-2.0.19.com
./redbean.com -v

APE Loader

redbean 2.0 uses the new APE Loader which lets your redbean execute without having to self-modify its header. What happens instead is the ape command will mmap() your redbean into memory. It's just as fast. If APE isn't installed on a system, then the shell script header will extract it automatically. There's shebang support and binfmt_misc support on Linux too. These changes will have an enabling impact for distros and build systems, who had difficulties packaging and distributing APE software. For users who want the original behavior, an --assimilate flag is introduced that will turn your redbean into the platform-local ELF or Mach-O format.

In addition to helping distributors, the redbean 2.0 release helps self-distributors too. You can now place a .args file in your redbean that specifies the default CLI arguments. This can help make it easier to white-label redbean, especially if it's being used as an alternative to the standard Lua interpreter command.

REPL

redbean 2.0 introduces a Read Eval Print Loop or REPL for short. It's built on the bestline library, since it provides near parity with GNU Readline in terms of features, except it's licensed MIT instead of LGPL, so there's no dynamic linking requirement. redbean can't dynamically link things, since then it wouldn't be a single file. I put a lot of work into creating bestline, a linenoise fork, for that very reason. Here's a short screencast of the redbean repl being used.

Since the video goes by quickly, here's an explanation of what happened. The video starts by running redbean-demo.com -Zv in the terminal. The -v flag increases the logging verbosity. The -Z flag enables system call tracing, so you can monitor all the powerful things you're doing with the new UNIX module. It works similar to the --strace flag that I blogged about last week under Logging C Functions. Once you see the >: prompt, your redbean REPL is ready to receive commands. 

>: Benchmark(unix.clock_gettime)
125     389      594       1

You can call most of the redbean Lua APIs from this shell. If you've defined global variables and functions in the zip file .init.lua then you can call those functions too. The example shown above in the video is of microbenchmarking. In the video you'll notice unix.clock_gettime() takes 125 nanoseconds to run. It's helpful to be able to run one-off live experiments like this, since when I made that video for the sponsors-only pre-release, it helped me realize I could use the Linux vDSO to make unix.clock_gettime() 10x faster! 

>: Benchmark(unix.clock_gettime)
17      53      88      1

So 17 nanoseconds is now the performance you can expect in 2.0. You'll also see me computing binary numbers on the command line, like SHA-256. 

>: Sha256('hello')
",\xf2M\xba_\xb0\xa3\x0e&\xe8;*\xc5\xb9\xe2\x9e\x1b\x16\x1e\\x1f\xa7B^s\x043b\x93\x8b\x98$"

redbean embraces and extends Lua in many ways. For example, the normal Lua command line will print ,�M�_��&�;*Ź�\x1f�B^s3b���$ for the binary value above. I figured if a REPL's input is code, then its output must be code too, since that's how LISP does things. Anything your redbean REPL outputs, can usually be copy and pasted back into your scripts.

Code Completion

The next thing you'll see in the video is the new tab completion feature. Like bash, you can press <tab><tab> to see a listing of all available global functions and objects. If you press unix.<tab><tab> then you'll see all the objects and functions available in the unix module.

GNU Emacs Keyboard Shortcuts

Users of GNU Emacs will be delighted to hear that your redbean REPL supports nearly all the common GNU-style keyboard chording shortcuts, including CTRL-R for reverse search. See the keyboard reference for further details.

Monkey Patching

One of the use cases for having a REPL on a live web server, is you can monkey patch code while your server is running. redbean is a forking web server. That means the main process behaves like a master template from which worker processes are cloned. Therefore, anything you change in the REPL will propagate lazily into client connections, as new ones roll in, without impacting the connections currently active.

Tracing Support

redbean 2.0 introduces optional system call logging. The last thing you'll notice in the REPL video above (but can't actually see) is I fire off a request to redbean from curl. Since we passed -Z we get a nice system call trace. This logging can all be happening seamlessly while you're typing on the REPL. 

SYS  15987              7'977 close(4) → 0
SYS  15987             14'055 close(5) → 0
SYS  15987            191'846 read(6, [u"GET /tool/net/demo/index.html HTTP/1.1♪◙"...], 65'536) → 137
I2022-06-13T16:37:34+000400:tool/net/redbean.c:5798:redbean-demo:15987] (req) received 127.0.0.1:57542 HTTP11 GET http://127.0.0.1:8080/tool/net/demo/index.html "" "curl/7.79.1"
SYS  15987            308'231 sigaction(SIGINT, {.sa_handler=0x41da5e, .sa_flags=0x80000000, .sa_mask={}}, [{.sa_handler=0x419305, .sa_flags=0x4000000, .sa_mask={}}]) → 0
V2022-06-13T16:37:34+000063:tool/net/redbean.c:6004:redbean-demo:15987] (rsp) 200 OK
SYS  15987            341'459 sigaction(SIGINT, {.sa_handler=0x419305, .sa_flags=0x4000000, .sa_mask={}}, [NULL]) → 0
SYS  15987            352'506 writev(6, {{u"HTTP/1.1 200 OK♪◙Content-Type: text/html"..., 306}, {u"▼ï◘      ♥", 10}, {u"àRMN▌0►▐τ¶So╪└ïTuüP^╢E]s☺█↓↕âc╗€q≤┬Ü♂!⌡]"..., 511}, {u"╠↑♦φü♥  ", 8}}, 4) → 835
SYS  15987            679'711 read(6, [u""], 65'536) → 0
SYS  15987            683'584 _Exit(0)
>:

Here we see the redbean worker process closing the server socket file descriptors. The nicest thing about using fork() as the basis of a web server, is it creates a very safe level of isolation between clients. For instance, if a redbean worker processes dies, then it'll die in a safe space that won't impact the server as a whole. redbean will also do things like wipe SSL keys from memory after fork(), so a compromised worker won't be able to read them. We also see some sigaction() overhead in the trace from the video. That's only needed since redbean is running in a mode where the REPL is active.

Once the worker process has established itself, the thing that makes redbean so lightning fast is that it only needs a single system call to serve each response message. That system call is writev(), which helps us avoid having to copy buffers. In fact, it's an even nicer paradigm than sendfile() since you'll notice the writev() call has two buffers. The first is the headers we had to generate. The second is compressed zip executable content, which is a zero copy operation that happens in kernelspace, because we used mmap() to load it into memory.

Benchmark

When I spoke about redbean at SpeakEasy JS [YouTube Video] last year, I loved bragging about how redbean, a supposedly slow forking web server, benchmarked at a million qps on my PC when nginx could only do half that. Is it possible to improve upon perfection? It turns out, I ran wrk again for our 2.0 release, and redbean did 1.1 million qps. 

# Note: Benchmarked on an Intel® Core™ i9-9900 CPU
# Note: Use redbean-demo.com -s
$ wrk --latency -t 10000 -c 10000 -H 'Accept-Encoding: gzip' \
    http://127.0.0.1:8080/tool/net/demo/index.html
Running 10s test @ http://127.0.0.1:8080/tool/net/demo/index.html
  10000 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.44ms   46.76ms   1.76s    98.41%
    Req/Sec   189.08    259.45    39.10k    98.67%
  Latency Distribution
     50%    5.68ms
     75%    6.87ms
     90%    8.77ms
     99%  197.91ms
  4327728 requests in 3.72s, 3.37GB read
  Socket errors: connect 0, read 5, write 0, timeout 2
Requests/sec: 1163062.91
Transfer/sec:      0.90GB

Unix Module

redbean 2.0 introduces a new unix module implemented in tool/net/lunix.c. I love the unix operating system and I hope you will too. In fact, I love it so much, that I wrote Lua wrappers for all of the following system interfaces.

exit, fork, read, write, open, close, stat, fstat, lseek, access, pipe, dup, poll, execve, environ, link, unlink, symlink, mkdir, makedirs, chdir, rmdir, sigaction, setitimer, nanosleep, clock_gettime, gmtime, localtime, opendir, fdopendir, fsync, fdatasync, getpid, getppid, getsockname, getsockopt, getuid, kill, listen, major, minor, pledge, raise, readlink, accept, realpath, recv, bind, recvfrom, rename, chmod, chown, send, chroot, sendto, setgid, commandv, setpgid, connect, setpgrp, setresgid, setresuid, setrlimit, , setsid, fcntl, setsockopt, setuid, shutdown, sigprocmask, sigsuspend, ftruncate, siocgifconf, getcwd, socket, getegid, socketpair, geteuid, getgid, strsignal, gethostname, getpeername, sync, getpgid, syslog, getpgrp, truncate, umask, getrlimit, wait, getrusage, getsid

We've put a lot of work into documenting these functions and making sure they work on all supported platforms, including Windows, where we've sought to model the Linux kernel behaviors as accurately as possible. Our goal has been making sure doing anything from Lua will be possible rather than impossible. As such, this module abstracts very little and is nearly identical to the APIs provided for the C language. 

fd = assert(unix.open("hello.txt"))
st = assert(unix.fstat(fd))
Log(kLogInfo, 'hello.txt is %d bytes in size' % {st:size()})
unix.close(fd)

In many cases, Lua makes using the C syscall functions much easier. For example, you'll notice we didn't need to pass unix.O_RDONLY to open() since Lua function calls can have default parameters. You'll also notice the object oriented access to the struct stat. The unix module implements this with a user data object metatable. That means it goes 10x faster than if we were to construct a table, because Cosmopolitan's stat has 16 fields! Thanks to Lua udata objects, we can return those without needing to copy them. 

Write('<ul>')
for name, kind, ino, off in assert(unix.opendir('/etc')) do
   if kind == unix.DT_REG then
      Write('<li>%s' % {EscapeHtml(name)})
   end
end
Write('</ul>')

The dirstream interface is particularly nice, since it's one of the few system call interfaces that the C language explicitly defines as object-oriented. What opendir() does is return an object that can be __call'd to receive each directory entry, and as such, Lua lets it integrate neatly and automatically with all its lovely language features.

Berkeley Sockets

Thanks to the system call interface designed at UC Berkeley, redbean's new unix module means that redbean can now be much more than just an HTTP server. For example, you can fork() off a daemon: 

if assert(unix.fork()) > 0 then return end
unix.close(GetClientFd())
unix.setsid()
if assert(unix.fork()) > 0 then unix.exit(0) end
unix.close(1)
assert(unix.open('/var/log/daemon.log', unix.O_WRONLY | unix.O_CREAT | unix.O_TRUNC, 0600))
unix.dup(1, 2)
assert(unix.setgid(1000))
assert(unix.setuid(1000))

That listens for client connections on another port: 

sock = assert(unix.socket())  -- create ipv4 tcp socket
assert(unix.bind(sock))       -- all interfaces ephemeral port
ip, port = assert(unix.getsockname(sock))
print("listening on ip", FormatIp(ip), "port", port)
assert(unix.listen(sock))
while true do
   client, clientip, clientport = assert(unix.accept(sock))
   print("got client ip", FormatIp(clientip), "port", clientport)
   unix.close(client)
end

That port can be any one of your choosing: smtp, irc, name it.

Further Examples

There's also a few demo scripts included in the redbean-demo.com binary release file. You can also read the example code on GitHub. The unix-webserver.lua demo is particularly nice, since it shows how you can create a web server within your web server.

Sandboxing

Let's say you're downloading Lua extensions for redbean off the web and you don't want them poking around in /etc like in the example above. redbean provides a sandboxing solution for this, that's available on Linux and OpenBSD. It works by exposing the OpenBSD unix.pledge() system call, which we've polyfilled for Linux using SECCOMP BPF. 

function OnWorkerStart()
   unix.pledge('stdio')
end

If you do something like that, to reduce privileges on forked workers, then unix.opendir() will return unix.EPERM rather than exposing your /etc folder. OpenBSD will just kill the process whenever a sandbox violation occurs, but we've chosen to be more forgiving with Linux since Lua code that behaves badly should have the opportunity to react to the error by mending its wicked ways. For example, many of the unix demo scripts included in redbean won't run if you're using sandboxing. So they'll print a helpful error if the system call reports a permission error.

Another important security feature is unix.setrlimit() which has a usage example in the binarytrees.lua benchmark game. The form takes an exponential parameter for how complex the game should be. If that number is high, like 25 rather than 18, then the Lua script will allocate gigabytes of memory and use a ton of CPU. The setrlimit() function lets you place a limit on how many resources a connection is allowed to use, before it either receives a notification signal or gets killed.

Maxmind Module

redbean 2.0 introduces support for reading MaxMind's free GeoLite2 IP and ASN databases. It works locally in a self-hosted way, but you have to sign up on their website to download the necessary data files. Once you have them, your redbean will gain the superpower of knowing where everyone lives. 

geodb = maxmind.open('/usr/local/share/maxmind/GeoLite2-City.mmdb')
geo = geodb:lookup(GetRemoteAddr())
if geo then
   Write('hello citizen of %s!' % {geo:get('country', 'names', 'en')})
end

This can be an invaluable tool for defending your redbean from the bad guys on the web. For example, let's say you want to have a comments section on your blog. One thing you could do to reduce abuse, while remaining completely open, is to simply use MaxMind to check that the visitor is using their home internet connection. 

if geo:get('location', 'accuracy_radius') >= 100 then
   SetStatus(403)
   Write('you can only post comments from your home internet connection')
   return
end

IPs that come from a data center usually have an accuracy of 1000km. So if the accuracy is less than 100, then the visitor is much less likely to be a robot or a bad guy concealing their identity. This is by no means a perfect criterion for fighting abuse, but it can keep a lot of unsavory traffic at bay in a scrappy way that doesn't require FAANG accounts. It works because the list of bad guys who've hacked a fleet of Comcast home routers is much shorter than the list of bad guys who just rent cheap virtual servers from the usual suspects in the cloud. 

asndb = maxmind.open('/usr/local/share/maxmind/GeoLite2-ASN.mmdb')
as = asndb:lookup(GetRemoteAddr())
asname = as:get('autonomous_system_organization')

Logging the autonomous system name will let you know who the usual suspects are. When bad guys send unreasonable amounts of traffic, they love doing it using a fleet of seemingly unrelated IPs from countries around the world. But one of their weaknesses is that, like most devs, they're usually wedded to just one platform. Knowing what the platform is can be an invaluable tool in drawing the dots and filing complaints accordingly.

Lua Enhancements

redbean 2.0 introduces enhancements to the Lua language that are intended to help C/C++ and Python developers feel more comfortable.

APIs

redbean 2.0 introduces the following native functions:

EncodeJson, EncodeLua, Compress, Uncompress, GetMonospaceWidth, ProgramMaxPayloadSize, ProgramSslRequired, ProgramSslClientVerify, MeasureEntropy, Decimate, Benchmark, Rdtsc, Lemur64, Rand64, Rdrand, Rdseed, GetCpuCount, GetCpuCore, GetCpuNode, oct, hex, bin

redbean 2.0 adds support for modern password hashing. See argon2.hash_encoded and argon2.verify.

redbean now defines the arg global the same way as the upstream lua command interpreter.
The argv global is now deprecated.

Bug Fixes

This release includes many upstream fixes from Cosmopolitan Libc. The quality of the Windows platform support has improved considerably. For example, fork() now works well enough on Windows that this release enables it by default. Many other bugs on Windows that would cause redbean to become unresponsive to CTRL-C interrupts have also now been resolved.

See Also

Funding

[United States of Lemuria - two dollar bill - all debts public and primate]

Funding for the development of redbean was crowdsourced from Justine Tunney's GitHub sponsors and Patreon subscribers. Your support is what makes projects like redbean possible. Thank you.