A transition to nginx - by necessity
So, nginx, what is it? It’s what all the cool kids are using in their Rails deployment stacks.
I don’t quite run with all the new trends in rails (i.e. I’m not a huge fan of REST, I don’t run edge for any of my projects), but this is a story how we were backed into a corner, and really didn’t have a any other choice (except Apache, but lets not go there).
In fact, this is actually an (kind of) argument for nginx as the premiere load balancing proxy / web server for your deployment stack, but first some background…
We had been running Pound to Mongrels for, well, months. Months and months. Eventually we threw in Apache to serve out static content (and surprise surprise, it made a massive difference to performance), yet under load Apache would spawn ever more processes, consume ever more memory and in general just annoyed the living daylights out of me.
We took lighttpd under our wing and threw Apache out on it’s arse. lighttpd was great, he’d hang around, do his job, and generally not take up too much room or make too much mess.
At some point, I got right fucked off with Pound (I can’t recall why) and decided I wanted to promote lighttpd to the role of proxy balancer / mongrel herder. And to which we did, and lighttpd did a great job. And the configuration file was small, comprehensible and forthcoming–the kind of thing you could read to your kids as a bed time story.
But then something happened–we started doing more business. People were downloading more and more files, and at the time we had used the seemingly benign send_file. As in, buffer the whole file into memory (I think) then pass it to lighttpd. As in, for some reason not free up that memory. As in, OOM killer killing off lighttpd at midnight. As in, Pingdom sending me a series of SMS’ while I’m in Newtown with Dylan & Co. pissed as a fart. As in, not the best kind of situation.
So send_file was sending our Mongrels rabid and we needed a solution, fast. I recall one of my reasons for moving to lighttpd in the first place was to use this so-called X-Sendfile thing. Hey, I heard it was great, right?
Turns out you can’t use X-Sendfile with lighttpd’s mod_proxy, so unless you’re running fastcgi (but you’re not, are you? that’s a good developer) you cannot leverage it.
lighttpd 1.5 (not yet production ready) has a brand-spanker called modproxycore, a complete rewrite of mod_proxy, with X-Sendfile support.
I couldn’t get X-Sendfile with 1.5 working on my local environment, I just fucking couldn’t get it working, I couldn’t figure out if it was this alledged bug in Rails or some other reason or I’m just a complete idiot but I could not get the fucking thing working. I spent like 3 hours on it. I even fixed the alleged rails bug, didn’t even submit a patch (so sue me–last time submitted a bug fix it was never applied–boo hoo). It still didn’t work.
We were prepared to run flashden.net on an experimental lighttpd 1.5, but alas, it wasn’t meant to be.
After some oogling I had a feeling now was the time to look into this nginx thing. I remember hearing people say things like “it is great, it’s really fast, you should try it!” but I dismissed it as more REST-like hype. And you know how I feel about REST. Well, you don’t. I’ll tell you some time.
What turned me off nginx initially too was that it was not just a load balancer, and a web server, but it was also a imap/pop3 proxy or something, and I thought “shit mate, what fucking else are you?”. But when you’re left raw with nowhere else to turn you kind of start looking places you’ve never considered looking before…
Thus my love affair with nginx began. Of all the things nginx claims to be, it has some unique-ish features; like it’s X-Accel-Redirect header, kind of like the venerable X-Sendfile but unique to nginx. This is what we’re looking for, so I took nginx out for the day to see if we’d get along.
It turns out we did! By the end of the afternoon we had updated our application code and had a bona-fide configuration file courtesy of this bloke along with some of our own junk config. nginx had officially moved in.
The end result? No more memory leaks due to send_file. A seemingly snappier site and the ability to go out and get as drunk as I like not having to worry about the site going down.
lighttpd will always have a place in my heart, yet nginx most surely is the new black and I can highly recommend it in a production setting… At least until lighttpd 1.5 finishes its therapy…
Are you still there? If you hang around for a while longer I’ll go into specific detail of, well, things that will actually contribute to you and your work, i.e. how to implement this crap. Yeah that’d be good. I’m sure you’re into that.