/robots.txt mechanism for crawler control is kind of a kludge, and works really badly when multiple sites are on the same server, for example members.aol.com. Another symptom is the fact that it's kind of hard to find the RSS feed for a web site. Well, we may be starting to address this issue over in the TAG ...