All About Search Indexing
Robots and Spiders - The title says it all -- an excellent article
covering everything you ever wanted to know about search engine crawlers and
how they work, with extensive links for more information.
- If you have access to your server logs, you can ferret out which spiders
or crawlers have visited your site, and exactly what they've fetched. Here's
how to do it.