Urlfetch redirects, an App Engine mis-documentation
So I’ve spent a couple evenings working on a little App Engine project and butting heads with the urlfetch module. What I’ve been trying to do is discover what the end location is for a 3xx redirect. I finally figured it out, to my chagrin, and thought I’d post it up in case anyone else is struggling with this minor problem. I thought this would be easy squeezy - in the urlfetch docs:
fetch() does not follow HTTP redirects. Instead, it returns the response object directly, with the status_code and headers set accordingly.
And yet, no matter what I tried (which wasn’t really all that much, given how nice and simple urlfetch is - really, it’s a nice interface) it was always returning the content of the final page. Finally, I had to bust out my non-existent Python skillz and track down the module involved. After some mucking (I’m on OSX with the remarkably awesome GoogleAppEngineLauncher) I found the file urlfetch_stub.py - it’s the stub fetcher for the development SDK since the live version on Google has a little bit that goes through their proxies. Pretty neat.
Tucked away in there is:
REDIRECT_STATUSES = frozenset([
httplib.MOVED_PERMANENTLY,
httplib.FOUND,
httplib.SEE_OTHER,
httplib.TEMPORARY_REDIRECT,
])
And then in a forever loop which keeps getting urls:
if http_response.status in REDIRECT_STATUSES:
url = http_response.getheader('Location', None)
Oh noes! So much for returning the location - it explicitly grabs up to 5 levels of redirection (if you get more than that it errors out) and tries its utmost to give you the final page. On a lark, I added this into my code just before a fetch:
urlfetch.REDIRECT_STATUSES = frozenset([httplib.SEE_OTHER])
And lo and behold the results contained my Location header. Of course, deploying this doesn’t actually work. Sigh. So alas it seems that for now, at least, no love for this problem.
Do not get me wrong, there’s something very fun about working with AppEngine - it really has a certain je ne sais quoi! I like working with Python, it’s no Perl :) but it’s got some nice features that I’m slowly getting the hang off. A very, very different mindset more of a “Do What I Say” instead of Perl’s “Do What I Mean”. App Engine as a framework is pretty easy to start with. The big thing for me, though is that there’s something really satisfying about deploying something and having it be live up on all that hardware at the push of a button. Anyone else playing out there?
