Google SEO is a delicate flower

So this blog does ok, it does better than I thought it would when I started out. Which is good because I need lots of encouragement to keep doing things, otherwise my various obsessions tend to peter out. I’ve noticed, from very early on that Google delivers a significant portion of my traffic, certainly at least 90% of my search engine driven traffic (according to enquisite) so while I don’t necessarily obsess over SEO activites, I see if there’s things I can do to help every now and again.

One thing I had noticed is that WordPress (that powers this here blog) creates rss feeds for the comments on every post. That didn’t seem like an overly good bunch of pages (close to 500!) to have in the index - I didn’t want anything confusing poor Google’s mind about what was important on my site. So I set out to get rid of this, initially I despaired because the robots.txt standard doesn’t allow wild cards and all these urls look something like /2007/01/21/the-article/feed/ - so the identifying feature is that /feed/ at the end. Fortunately it turns out the bright minds at Google figured out that people might want to do that and added in some patterns to their robots.txt parsing!

In my searching for this stuff, I discovered that lots of other people had this thought too, so much so that they already created a model robots.txt file to try and use - so I grabbed it:

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$

User-agent: *
Disallow: /wp-

They threw in some stuff that I hadn’t thought of, so I was excited. Using the Webmaster Tools dashboard I checked to make sure those would all work - it has a convenient robots.txt tester which allows you to test how the Googlebot will respond to your robots.txt and also shows you when it’s pulled your new one. Everything looked good and I pulled the trigger.

I found this tool to calculate your Supplemental Index Ratio - basically the Google Supplemental Index Ratio is a measure of how many of your pages Google likes and how many it doesn’t - the lower the ratio the better. Like all SEO measures, there’s dispute over how interesting this number is. At any rate when I started this li’l project I was around 63% with 610 in Google’s main index (the good index). Now? A week or so later, I’m at 80% with about 300 in the main index.

Sad! This has also corresponded with a lowering (not drastic, but clearly noticeable) of Google traffic to the site. It’s hard to say what has happened. In the Webmaster dashboard it tells me that as of now 286 files have been blocked by robots.txt and they’re all the right ones (it conveniently provides a list for you!). But, even the ones I see in that list still come up the google search for my site. Hopefully, it just takes a little while for the index to catch up.

I’m hoping that like many SEO initiatives that this is a worse before it gets better situation where doing the right thing often confuses Google and hurts your score for a little while but once Google settles in, gets used to the new waters, brings you back better than before. Search engine optimization seems much more art than science so you take the good with the bad. And there you have the facts of life.

Sep 11th, 2007 · Comments RSS · · · · · Tags: google · navel gazing · seo