How many people actually have experience to help Twitter?
You know, there’s a lot of bitching about Twitter downtime going around, not undeservedly so. The service is experiencing significant growth pains and is down more than I’ve seen any other popular web service go down. Bitching is one thing, it’s one of the main joys of life - complaining and whining and what not, I do it all the time about everything. Ask my wife.
But the twitter whining seems to have started all sorts of new threads. The most reasonable of which is that Twitter should be more transparent about their downtime. I’m down with that - transparency in a company is great from a user perspective. But then come all the calls to open source the code base, open source the architecture, ask the users to see who can help them scale out.
What? Twitter’s out there raising millions and millions of dollars, do people really think that they aren’t finding the best people they can find to help them out? Do people really think that they themselves are going to have some great idea that’s going to solve these problems for Twitter? First of all, let’s take a look at the realm of people that have actual experience scaling a dynamic site to the level of a Twitter like website. What is the realm of people who have worked on the architecture of a site like that?
Beyond that, it seems to me that twitter is a site quite unlike any other site. Obviously every site is different, but Twitter, to me, seems even more different. They have tons of small bits of data with new data pouring in faster than you can shake a stick and unlike IM all that data is stored. Forever. And they’re pouring out data differently for every single user and many of those users are updating every minute or every few minutes, 24 hours a day, 7 days a week. How many people have worked on things like that at the scale that Twitter is operating?
It seems to me that sure there’s two things that could be offered. First is good basics - Twitter perhaps has a basic architecture, a legacy from their humble beginnings, that no longer scales. There’s are a lot of scaling fundamentals that need to be put into the architecture. Well, they are fundamentals for a reason, they’re reasonably well understood now so Twitter in rearchitecting their site (Ruby or no) will implement these anyway, they don’t need to be told. They’ve probably spent too long trying to make their flawed architecture work but they’re now at the point where they’re looking into a whole new thing - they got it, they don’t need anyone they didn’t hire to tell them.
The other thing that would come of this are good theoretical ideas on how things should scale for Twitter in particular. The problem is, I suspect, that as with everything there’s tons of devils in all the details and when you’re running at scale there are tons of details resulting in an exponential explosion of devils. Getting things right is an iterative process of starting with some good theories, putting them into practice and seeing what actually happens and tweaking and refactoring from there. So, no, I don’t think opening the floodgates to the public to input their great ideas from all those guys who ran that ruby website that did that thing is going to help Twitter one iota, it will be detrimental because they’d have to spend their time trying to explain to everyone why Awesome Idea #1252 isn’t actually that awesome instead of just trying to figure out what’s happening and fix it.
I’m just saying.
