Publisher's Synopsis
Take Web crawler one step further. There has never been a Web crawler Guide like this.
It contains 183 answers, much more than you can imagine; comprehensive answers and extensive details and references, with insights that have never before been offered in print. Get the information you need--fast! This all-embracing guide offers a thorough view of key knowledge and detailed insight. This Guide introduces what you want to know about Web crawler.
A quick look inside of some of the subjects covered: HTTrack, Digital time capsule - Wayback Machine, Nutch - Scalability, OAI-PMH - Uses, Googlebot, Secure server - Limitations, User agent - User agent identification, Social bookmarking - Comparison with search engines, Robots Exclusion Standard - History, Lynx (web browser) - Web design and robots, Spamdexing - Cloaking, The Internet Archive - Wayback Machine, Web directories, Spokeo - Technology, Library for WWW in Perl - History, Semantic Web - Current state of standardization, Ajax (programming) Drawbacks, Lese majeste in Thailand - Internet blocking measures, POST (HTTP) - Affecting server state, TkWWW - The TkWWW Robot, Email address harvesting - Methods, HTTP Secure - Limitations, Webserver - Overview, Robots.txt, Spamdexing - Page hijacking, Web crawling - Examples, HPCC - Introduction, Nutch - Features, IRC - Bots, Alexa.com - Operations and history, Canonical link element, Web spider - Open-source crawlers, Digital library - Searching, Video search, Open Directory Project Maintenance, Video search engine, Index (search engine) - Challenges in parallelism, Heritrix, Web crawler - Academic-focused crawler, Wget - Recursive download, DARPA balloon - Tenth-place strategy, HTML element Document head elements, Web archiving - Remote harvesting, Distributed web crawler, Site map - XML Sitemaps, and much more...
