CSE341 -- Assignments -- Java warmup

Due 26 April 2000.

This is a practice assignment to give you some experience with Java.

Submission directions: Please turn in both printed and electronic copies of your code. The print submission is due in class on the due date above. You may e-mail the electronic copies to Keunwoo (klee@cs) anytime the same day. For full credit, you must follow the submission guidelines.

• Overview
• Details and definitions
• Tips

Overview

Web crawlers (or "spiders") are programs that index the web by starting with some known URLs, finding pages that these pages point to, then pages these pages point to, and so forth, indexing as they go.

Write and test a Java program that accepts a URL and a number n from the command line, then follows n random links, starting from the given URL, printing each URL it visits. For example, here is a sample invocation and output:

orcas% java WebWorm http://www.cs.washington.edu/education/courses/341/CurrentQtr/ 5

Starting crawl from http://www.cs.washington.edu/education/courses/341/CurrentQtr/
 href="http://www.cs.washington.edu/homes/borning/"
 href="http://www.cs.washington.edu/education/courses/510/99wi/"
 href="http://www.uwtc.washington.edu/courses_and_programs/courses/tc_517/Default.htm"
 href="http://www.uwtc.washington.edu/"
 href="http://www.uwtc.washington.edu/other_resources/Default.htm"

If you are unfamiliar with HTML, the UW has list of resources for learning HTML and the web that you might find useful.

Details and definitions

For the purposes of this assignment, a link is defined as a string of text that:

• Begins with the string <a href=
• Ends with the first > character after the href=
• Does not contain the strings mailto:, ftp:, news:, or gopher: (these exceptions are so that you don't have to handle non-HTTP links.)

The target of a link is the part that goes between the href= and the closing > symbol. The target may optionally be encapsulated in quotes; you should strip these before attempting to follow the target. Targets may be absolute (they may contain a full URL, e.g. "http://www.cs.washington.edu") or relative (they only contain a relative path, e.g. "../perl/index.html" and "java/example.html"). Your crawler must be able to handle both cases. You do not need to parse non-HTML targets.

Your program should basically consist of a loop that fetches the page at a given URL, extracts and parses all the links (accounting for relative URLs), selects one at random, and then fetches it.

You do not have to handle links that span multiple lines. Also, if you encounter a dead end (a page with no links), your program should exit and let the user know.

For extra credit, you can adapt your crawler to do something reasonable when it encounters circular chains of URL references.

Tips:

• A Java program to read the contents of a Web page is on page 345 of the text; you can get it here as well if you want to avoid retyping it. I'd use this program as a starting point. The code includes an exception handler (the try - catch stuff). You can use this part of the program unaltered (and without worrying about completely understanding it at this point) -- we'll talk about exceptions in more detail shortly in the lectures, and they are also covered in Chapter 16 of the text.



• There are some useful methods in class String for searching a string for a substring, extracting a substring, and so forth.



• If you read the contents of the URL line by line (as in the ReadURL program), you don't have to deal fully with the problem of links that span multiple lines in the html file -- if you find a string <a on an input line with no matching closing bracket, then just print out as much of the link as you find on that line. (The html for the CS department home page has such a link.) If you are bothered by the unaesthetic nature of this solution, and want to handle this problem correctly, I'd read characters directly from the BufferedReader rather than reading a line at a time -- but you don't need to do this.