Web crawler
INTERNET BOT THAT SYSTEMATICALLY BROWSES THE WORLD WIDE WEB, TYPICALLY FOR THE PURPOSE OF WEB INDEXING (WEB SPIDERING)
Web spider; Search engine spider; Search engine spiders; Webcrawler; Web crawling; Search engine robot; Search engine robots; Web-crawler; Search bot; Web Crawler; Spidering; Web crawlers; Search robot; Crawl site; Webcrawlers; Scutter; Web scutter; Spiderbot; Spiderable; Site crawler; Spider operating system; Automated agent; Spider bot; FAST Crawler; RBSE; Automatic indexer; Searchbot; Semantic focused crawler; Search-engine spider; Internet spider; List of web crawlers
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).