|
|
|
|
- The best overview of a crawler:
Mercator: A Scalable, Extensible Web Crawler,
Allan Heydon and Mark Najork, Compaq SRC, June 1999. [Req]
- What order a crawler should use when following links?
Efficient Crawling Through URL Ordering,
Junghoo Cho, Hector Garcia-Molina and Lawrence Page, Stanford University, 1998.
- Topic-specific crawling:
Focused Crawling: A New Approach To Topic-Specific Web Resource Discovery,
Soumen Chakrabarti, Martin van den Berg and Byron Dom, Elsevier Science B.V., 1999.
|