Dear readers, honesty (and the story I’m about to tell) require that I spill the beans: there is lots of stuff here on ongoing that you can’t see. Since I have a pretty good writing environment, I compose lots of little pieces of one kind or another and then “semi-publish” them; they’re out there on the Web at an address that looks a lot like that of the fragment you’re now reading, but there are no pointers to them; security by obscurity, but good enough for my purposes. Last night I semi-published something and emailed a few people asking them to look at it. A couple hours later, I wondered if they had, and checked the server logs. I saw myself (twice, I’d corrected it), and... the Googlebot. What the hell? Did I accidentally press “publish”? No. Bafflement. Is my browser telling Google where I’m going?!? Unlikely... ah! Even semi-published pieces have the ads, which are a Javascript callback to somewhere in the Googleplex. So, that’s what’s happening... when AdSense displays on a page, at least sometimes, it tips off the robot army. So anyone who’s running AdSense gets indexed first & fastest. I can’t prove it, but it’s the simplest explanation, and it makes all sorts of sense. [Update: If you look real closely at that robot (for geeks, at the “User-Agent” field), it’s not quite the same as the normal googlebot; apparently this beast is just reading the text of the page to figure out what contextually appropriate ads to display. Thanks to all the people who wrote to point this out.]
