What does a Crawler look like?!
Search Engine Basics: Crawling, Indexing & Ranking - Totally Communications
Having watched the recent episode of Halt and Catch Fire, I may now have a slight fascination with Web Spiders. What would they graphically look like?
Crawling
Crawling is the process by which search engines discover updated content on the web, such as new sites or pages, changes to existing sites, and dead links.
To do this, a search engine uses a program that can be referred to as a 'crawler', ‘bot’ or ‘spider’ (each search engine has its own type) which follows an algorithmic process to determine which sites to crawl and how often.
http://www.totallycommunications.com/latest/search-engine-basics-crawling-indexing-ranking/