Unencumbered by Infrastructure

🚗 If you don’t want to read the whole thing, I have a working live editing Rust playground embedded at the bottom of this post

Serverless as a paradigm has been a fad for a long time. You hear the tales of infinite scalability, of chips in the cloud gracefully accepting your bits when you need them, only to die back down when you don’t. You drink the kool-aid and deploy some lambdas. You are happily chugging along with your army of lambdas when one day you realize that you oh-so-badly want to coordinate multiple lambdas to, oh I don’t know, rate limit a particular execution. The simple units of compute you were promised have morphed into a nine-headed Medusa of a distributed system.

Apart from inevitable accidental complexity, I’ve found it quite awful to develop with serverless. It’s hard to run things locally in a tight loop, and deploying to STS to debug your infra is hellish. I’ve also never found the actual process of setting up serverless infrastructure to spark more joy than spinning up an equivalent EC2 instance. Sure I have to deal with AMIs being annoying, but in return I get full access to my infrastructure. I’m not forced to write in a specific language (like Cloudflare workers) or write Dockerfiles (like Fly). So maybe that’s my real issue - bootstrapping a simple app on serverless infrastructure just doesn’t feel easier than just spinning up a box on Hetzner and throwing Traefik with my own Docker containers at the problem.

Recently however, I needed access to GPUs. I’m working on a fun little Stable Diffusion based side project (yes, yes, me and everyone else), and I could no longer use my Hetzner strategy because I have not gotten VC funding to perma-run GPUs on the cloud. I thought about buying a PC to run things locally and expose it over the internet with DuckDNS, but I don’t trust my security posture enough to know that I’m not going to expose my network to a friendly neighborhood hacker.

Enter Modal - a way to run GPUs in a serverless manner. Not only do you get to run GPUs that are charged by the second through technology that feels like magic, I never have to write a Dockerfile to do things. While I started using Modal to experiment with Stable Diffusion, I realized that the development experience being so magical made it equally good to deploy any workload.

This post is not yet sponsored by Modal - although I’d love if anyone paid me to market to my 2 readers from Russia every month!

To test out Modal’s capabilities I decided to start simpler. I’m working on a mini-project to extend Sandpack to support every language under the sun so I can write beautiful codeblocks that are live-editable and runnable. My first language extension is going to be Rust - I’ve learnt so much about the language at work, and would love to be able to write down some of my learnings.

Nobody has an easy way to compile Rust in the browser yet, so I knew I’d have to set up a simple backend that takes in the files from my codeblock and compiles it. The problem with setting this up on Hetzner is I still wasn’t confident in my security posture. My Hetzner instance has some sensitive data (a Namecheap API key, a read-only Github key, some data for the apps I host on it), and compiling arbitrary Rust code over a public endpoint seemed like a surefire way to leak all that information.

While laying in bed with my last 2 brain cells recovering from jet lag, it hit me - Modal (and any serverless provider) has already built a secure execution environment for untrusted user code. I can run my function on Modal, put some basic timeouts and cost restrictions, maybe slap on some DDoS protection and be completely free of the burden of someone catastrophically gaining access to my host. The worst they can do is spend some money, and as long as Modal’s spend limits work, I can just set them to an amount I’m willing to lose. (They also give you $30 of free credits which goes a really long way when you’re charged by the second). There were some nuances in setting things up:

• the default Function execution allows users to access your Modal API token, so you have to use their sandboxing.
• Sandboxes don’t let you stream output, so there isn’t feedback to the user as the code is being compiled.
• The cold start time is pretty slow (about 45 seconds), and I haven’t looked into the cost implications of keeping warm containers around.
• I wasn’t able to figure out how to mount an NFS to the sandbox from my remote function, so I did something potentially insecure.

For the most part though, things just worked. After spending way too much time reconfiguring my blog to work with Tailwind so I could style one div below not super well, I finally have live codeblocks for Rust!

I’m going to spend a lot more time polishing things up (especially the looks and ergonomics), but I’m pretty happy with this first iteration!