High Scalability Blog on Stack Overflow

by Jeff Atwood

on July 17, 2009

It was quite an honor to see that the High Scalability Blog posted an entry on Stack Overflow!

We referred to the HSB, and its exhaustively detailed information about how other websites handle scaling, many times during the course of Stack Overflow development. And I've cited it myself when researching what we think is the largest public same-stack (Microsoft) website on the internet, Plenty of Fish. It's an excellent resource.

I don't know if Stack Overflow has that much traffic relative to many of the other truly giant public websites profiled on the HSB. You can see our public Stack Overflow traffic stats at Quantcast if you're curious. Still, it's great to be able to give back to the community and help document our own process of scaling our little corner of the web.

That said, I agree with the overall conclusion that Todd Hoff reaches:

If you need to Google scale then you really have no choice but to go the NoSQL direction. But Stack Overflow is not Google and neither are most sites. When thinking about your design options keep Stack Overflow in mind. In this era of multi-core, large RAM machines and advances in parallel programming techniques, scale up is still a viable strategy and shouldn't be tossed aside just because it's not cool anymore. Maybe someday we'll have the best of both worlds, but for now there's a big painful choice to be made and that choice decides your fate.

Scaling up is definitely a viable solution, as both Plenty of Fish and we can attest.

Like all of Todd's pieces, it is exhaustively researched and documented, and well worth your time to read. I was a little stunned how thorough it was, actually -- I doubt anyone outside our core development team has thought about our design and scaling this much!