spider

Synonymous with a crawler, this is a program that searches the Internet and attempts to locate new, publicly accessible resources, such as WWW documents, files available in public FTP archives, and Gopher documents.

Also called wanderers or bots, spiders contribute their discoveries to a database that Internet users can search by using a search engine. Spider technology is necessary because the rate at which people are creating new Internet documents greatly exceeds any manual indexing capacity (although search directories prefer the manual approach).