GoPlan interview – EC2
You may remember a bit ago that I reviewed newcomer GoPlan to Basecamp. I was happy that they got in touch with me after that review to let me know that their future plans were in line with my own needs. I’m psyched to hear it because Basecamp, as good as it is, doesn’t seem to be adding any significant new functionality at this point and there’s things that I find it lacks. I was also very interested to discover that GoPlan was using Amazon’s EC2 and S3 services. I thought I’d give it a shot and asked if they’d mind a short interview with them about how working on EC2 was and with great graciousness they agreed to do it and put me in touch with one of their developers, Tiago. I tried to keep the interview short as he’s quite busy but despite a lack of time he answered very completely and fairly.
The interview after the jump!
#: Was Goplan always conceived as a project to be run using Amazon’s EC2 and S3?
Tiago: Goplan was not initially designed to use EC2 or S3, in fact, for a few months it was running on TextDrive.
#: What was it that made you shift your plans to get it running in the compute cloud?
Tiago: The switch to EC2 came when we needed to scale the application. Not only was it cheaper to run it on EC2, it was also easier to scale during the launch. The ability to launch more instances on demand was very useful when we were featured on TechCrunch, digg, etc. Using S3 was a natural consequence of our move to EC2.
#: How did you get in touch with Amazon in the first place, did they reach out to you?
Tiago: The EC2 beta invite was given to us by Amazon when we asked for one.
#: What were the key stumbling blocks towards moving GoPlan to EC2 and S3? Did a lot of code need to change to work in that environment, new tools created to get things onto S3 or was it pretty straight forward to migrate from your existing rollout?
Tiago: Moving to EC2 and S3 wasn’t that hard. When we got the EC2 account, we decided to move Goplan to it: even though we only had a few dozen users we were having some problems with our TextDrive account.
The main disadvantage that EC2 has compared to other solutions is the lack of permanent storage, that forced us to change the file storage system to use S3 instead of just keeping them on the filesystem. The are a variety of ruby libraries that allow you to use S3 and we decided to go with s33r by Elliot Smith (which was actually very helpful) but we’ll probably have to move to aws-s3 as s33r is now deprecated.
Another problem that arose from this was the database. Initially we were running the database on a single EC2 instance and performing hourly backups to S3 but this solution didn’t assure us that we wouldn’t have data loss if something happened, so we decided to change a number of things. We’re still running our PostgreSQL database on a single instance but we moved to a WAL approach to assure that in an event of data loss, we can recover it using the logs. That being said we never had any problems with EC2, our database instance has a 6 months uptime (which is remarkable for a Xen based system).
At the moment the database is far from being a bottleneck even though the instances are moderate machines (similar to a Xeon 1.7ghz with a 1.7gb of ram). We’re using Munin to monitor all our instances so if our DB instance starts having performance issues we’ll be able to fix it.
#: Recently, there’s been a little bit of buzz recently about the scalability of Rails. Obviously, GoPlan doesn’t have the same scale as Twitter, but does EC2 alleviate any of the problems that may come from Rails’ potential bottlenecks? I guess I’m curious to know how you scale a database on EC2 since my understanding is that one compute unit is the equivalent of a moderate machine. Are you still running off a single database or have you moved to a more complex db configuration?
Tiago: The Twitter issue was clearly over exaggerated. They started having a lot
of traffic overnight and when they started to work on it they hit the database performance wall. With a standard configuration, Rails won’t allow you to use multiple databases, this however can be address in
several ways: by changing that behaviour for read-only queries (it wouldn’t even be a day’s worth of work), by using a middleware solution (such as SQL Relay) or by addressing that at the database level (using Master-Master replication). MySQL has internal replication solutions, PostgreSQL has PGCluster (I have some experience using it) and there is always the option of using more complex (and expensive) solutions such as Oracle databases.
Scaling the Rails application layer is pretty much straight forward: cache everything you can (which we do extensively on Goplan) and add more hardware if necessary. Stefan Kaes’ blog (Rails Express) is a good
starting point for Rails scalabity issues.
#: Have you yet had a situation where you had a usage spike and needed to rollout additional computing power and it was like a 30 minute exercise to get as much compute power as you needed?
Tiago: In our launch week we had a need to add more instances than we usually run on. As we started getting hit by both TechCrunch and Digg we thought it would be best to add more instances or the user experience would seriously degrade. Adding those new instances was 30 minutes work as our setup isn’t “perfect” as of yet, it required granting firewall and database access, adding the new instances to the load balancer, etc. However, there are some people working on solutions that add new instances on demand so you might want to take a look at their solution (WeoCEO).
Our biggest issue with EC2 so far has been that some people (mostly from the US) claim there is a latency issue and we’re trying to determine what’s causing this and how can we address it.
#: Are there any stats you can give me to get a sense of the scale that GoPlan’s currently at? I understand completely if that information can not be made public.
Tiago: I can tell you we have a maximum of 10.000 full pageviews per day, averaging at about 8000. At the moment I can’t give you statistics on ajax and rss requests as I don’t have them myself but I’ll look into this over the next few days. I can also tell you that we have an average of 10 pages per visit per user.
As far as stats go, approximately half of our traffic is US based and the remainder is spread across Canada, Europe and Asia.
#: I just want to think Tiago and Fred for taking the time to talk to me and organize this little interview. I’m definitely going to be watching how GoPlan evolves and if it has any impact on Basecamp’s development, thus far Basecamp has been very idle and I’ve heard that they aren’t so great with the customer service. GoPlan seems to be moving quickly and obviously seems to care about their users (and even non-users!). And I won’t lie and say I’m not a little jealous that they get to play with EC2, I’d love to be able to play around in that space to see what would be appropriate there or not.