CSE143 Notes for Wednesday, 4/18/12

I said that in the next programming assignment we are going to return to being clients of Java's built-in collection classes. So we'll once again be discussing sets and lists and also a new kind of structure.

I began by asking how we could write a program that would count the number of unique words in an input file. I had a copy of the text of Moby Dick that we looked at to think about this. I showed some starter code that constructs a Scanner object tied to a file:

        import java.util.*;
        import java.io.*;
        
        public class WordCount {
            public static void main(String[] args) throws FileNotFoundException {
                Scanner console = new Scanner(System.in);
                System.out.print("What is the name of the text file? ");
                String fileName = console.nextLine();
                Scanner input = new Scanner(new File(fileName));

                while (input.hasNext()) {
                    String next = input.next();
                    // process next
                }
            }
        }

Notice that in the loop we use input.next() to read individual words and we have this in a while loop testing against input.hasNext().

So how do we count the words? Someone suggested that a set would be the perfect structure to solve this problem. It eliminates duplicates, so it will keep track of how many different words there are. So we changed the code to be:

        Set<String> words = new TreeSet<String>();
        while (input.hasNext()) {
            String next = input.next();
            words.add(next);
        }
        System.out.println("Total words = " + words.size());

Here is a sample log of execution:

        What is the name of the text file? moby.txt
        Total words = 32019

One limitation of this version is that it pays attention to capitalization. So it considers the words "whale" and "Whale" and "WHALE" to be different words. To fix that, we modified the code to read in a word so that it converts it to its lowercase equivalent:

        String next = input.next().toLowerCase();

It didn't make much difference, as we saw from this execution:

        What is the name of the text file? moby.txt
        Total words = 30368

Someone pointed out that we still haven't dealt with punctuation. It is considering "whale" and "whale." and "whale," to be different words. I didn't want to deal with it in this program, but I mentioned that the chapter 10 case study discusses this and shows you how to configure the Scanner so that it ignores those punctuation characters.

This program counts the number of unique words, but not the counts of the individual words. To keep track of word counts, we could use many different structures. For example, we could have a List of words and a List of counts where element "i" in one corresponds to element "i" in the other. This approach is often described as "parallel arrays." It's not a very object-oriented approach because we really want to associate the word with its counts rather than have a structure that puts all the words together and another that puts all the counts together. Or we could make a class for a word/count combination and then have a List of that. But Java gives us a better alternative. The collections framework provides a data abstraction known as a map.

The idea behind a map is that it keeps track of key/value pairs. In our case, we want to keep track of word/count pairs (what is the count for each different word). We often store data this way. For example, in the US we often use a person's social security number as a key to get information about them. I would expect that if I talked to the university registrar, they probably have the ability to look up students based on social security number to find their transcript.

In a map, there is only one value for any given key. If you look up a social security number and get three different student transcripts, that would be a problem. With the Java map objects, if you already have an entry in your map for a particular key, then any attempt to put a new key/value pair into the map will overwrite the old mapping.

We looked at an interface in the Java class libraries called Map that is a generic interface. That means that we have to supply type information. It's formal description is Map<K, V>. This is different from the List and Set interfaces in that it has two different types. That's because the map has to know what type of keys you have and what type of values you have. In our case, we have some words (Strings) that we want to associated with some counters (ints). We can't actually use type int because it is a primitive type, but we can use type Integer.

So our map would be of type Map<String, Integer>. In other words, it's a a map that keeps track of String/Integer pairs (this String goes to this Integer). Map is the name of the interface, but it's not an actual implementation. The implementation we will use is TreeMap. So we can construct a map called "count" to keep track of our counts by saying:

        Map<String, Integer> count = new TreeMap<String, Integer>();

The most basic methods in the map interface are the ones that allow you to put something into the map (an operation called put) and to ask the map for the current value of something (an operation called get).

I asked what code we need to record a word in our map the first time we see it. Someone suggested using the put method to assign it to a count of 1. So our loop becomes:

        Map<String, Integer> count = new TreeMap<String, Integer>();
        while (input.hasNext()) {
            String next = input.next().toLowerCase();
            count.put(next, 1);
        }

This doesn't quite work, but it's getting closer. Each time we encounter a word, it adds it to our map, associating it with a count of 1. This will figure out what the unique words are, but it won't have the right counts for them.

I asked people to think about what to do if a word has been seen before. In that case, we want to increase its count by 1. That means we have to get the old value of the count and add 1 to it:

        count.get(next) + 1

and make this the new value of the counter:

        count.put(next, count.get(next) + 1);

So we have two different calls on put. We want to call the first one when the word is first seen and call the second one if it's already been seen. Someone suggested using an if/else for this. The only question is what test to use. The Map includes a method called containsKey that tests whether or not a certain value is a key stored in the map. Using this method, we modified our code to be:

        Map<String, Integer> count = new TreeMap<String, Integer>();
        while (input.hasNext()) {
            String next = input.next().toLowerCase();
            if (!count.containsKey(next)) {
                count.put(next, 1);
            } else {
                count.put(next, count.get(next) + 1);
            }
        }

The first time we see a word, we call the put method and say that the map should associate the word with a count of 1. Later we call put again with a higher count. And we keep calling put every time the count goes up. What happens to the old values that we had put in the map previously? The way the map works, each key is associated with only one value. So when you call put a second or third time, you are wiping out the old association. The new key/value pair replaces the old key/value pair in the map.

Then we talked about how to print the results. Clearly we need to iterate over the entries in the map. One way to do this is to request what is known as the "key set". The key set is the set of all keys contained in the map. The Java documentation says that it will be of type Set. We don't have to really worry about this if we use a for-each loop. Remember that a for-each loop iterates over all of the values in a given collection. So we can say:

        for (String word : count.keySet()) {
            // process word
        }

We would read this as, "for each String word that is in count.keySet()..." To process the word, we simply print it out along with its count. How do we get its count? By calling the get method of the map:

        for (String word : count.keySet()) {
            System.out.println(count.get(word) + "\t" + word);
        }

I didn't try to print all of the words in Moby Dick because it would have produced too much output. Instead, I had it show me the counts of words in the program itself. Obviously for large files we want some mechanism to limit the output. At that point I passed out the handout with my commented solution. In that version, I include some extra code that asks for a minimum frequency to use. We ran that on Moby Dick and saw this list of words that occur at least 500 times:

        What is the name of the text file? moby.txt
        Minimum number of occurrences for printing? 500
        4571    a
        1354    all
        587     an
        6182    and
        563     are
        1701    as
        1289    at
        973     be
        1691    but
        1133    by
        1522    for
        1067    from
        754     had
        741     have
        1686    he
        552     him
        2459    his
        1746    i
        3992    in
        512     into
        1555    is
        1754    it
        562     like
        578     my
        1073    not
        506     now
        6408    of
        933     on
        775     one
        675     or
        882     so
        599     some
        2729    that
        14092   the
        602     their
        506     there
        627     they
        1239    this
        4448    to
        551     upon
        1567    was
        644     were
        500     whale
        552     when
        547     which
        1672    with
        774     you

One final point I made about the Map interface is that you can associate just about anything with just about anything. In the word counting program, we associated strings with integers. You could also associate strings with strings. One thing you can't do is to have multiple associations in a single map. For example, if you decide to associate strings with strings, then any given string can be associated with just a single string. But there's no reason that you can't have the second value be structured in some way. You can associate strings with arrays or strings with Lists.

In the final version that I passed out as a handout, I made one minor change. There is an interface known as SortedMap. When you know that you want to require a map to have its keys sorted, it is more appropriate to use that interface. Every SortedMap is a Map, but not every Map is a SortedMap. So the program changes the declaration to:

        SortedMap<String, Integer> count = new TreeMap<String, Integer>();

Then I mentioned that I wanted to explore a sample program that will constitute a medium hint for the programming assignment. We will begin looking at the program in this lecture and finish it up in the next lecture.

The sample program involves keeping track of friendships. You could think of it as keeping track of Facebook friends. One of the first questions that comes up is how do we represent friendships? For example, are friendships bidirectional? If person A is friends with person B, does that mean that person B is friends with person A? For our purposes, we will assume the answer is yes. If we were trying to represent something like "is attracted to", then we'd come to a different conclusion, but for friends, just like on Facebook and other social networking sites, friendship goes both ways.

I said that a good way to visualize friendships is to draw a graph in which each person is represented with a node (an oval) and each friendship is represented by an edge connecting two nodes (a line drawn between two ovals). I am using a program called Graphviz, which is an open-source graph viewer.. For example, here is a sample friendship graph:

This information is stored in a file with lines that list pairs of friendships, as in:

        graph {
            Ashley -- Christopher
            Ashley -- Emily
            Ashley -- Joshua
            Bart -- Lisa
            Bart -- Matthew
            Christopher -- Andrew
            Emily -- Joshua
            Jacob -- Christopher
            Jessica -- Ashley
            JorEl -- Zod
            KalEl -- JorEl
            Kyle -- Lex
            Kyle -- Zod
            Lisa -- Marge
            Matthew -- Lisa
            Michael -- Christopher
            Michael -- Joshua
            Michael -- Jessica
            Samantha -- Matthew
            Samantha -- Tyler
            Sarah -- Andrew
            Sarah -- Christopher
            Sarah -- Emily
            Tyler -- Kyle
            Stuart -- Jacob
        }

Then I demonstrated what the Friends program is supposed to do. It is supposed to use this data to find how far one person is from another. So starting with a given person, it finds that person's friends, then the friends of those friends, then the friends of the friends of the friends, and so on. It reports how far it has to go to find a connection and if it runs out of people, it simply reports that the connection couldn't be found.

For example, here is a sample execution using our data file for finding the connection between Stuart and Ashley:

        Welcome to the cse143 friend finder.
        starting name? Stuart
        target name? Ashley
        
        Starting with Stuart
            1 away: [Jacob]
            2 away: [Christopher]
            3 away: [Andrew, Ashley, Michael, Sarah]
        found at a distance of 3

It finds that Stuart has one friend (Jacob). And that friend has one friend (Christopher). And he has 4 friends, including Ashley. So the program reports that it found Ashley is 3 away.

Here is a sample execution where the connection is not found, asking for a connection between Stuart and Bart:

        Welcome to the cse143 friend finder.
        starting name? Stuart
        target name? Bart 
        
        Starting with Stuart
            1 away: [Jacob]
            2 away: [Christopher]
            3 away: [Andrew, Ashley, Michael, Sarah]
            4 away: [Emily, Jessica, Joshua]
            5 away: []
        not found

The program goes two levels farther than it did before, finding that it runs out of people when it gets 5 away from Stuart. At that point it knows that there is no connection between Stuart and Bart.

We looked at one more example that involved a fairly long chain:

        Welcome to the cse143 friend finder.
        starting name? Bart  
        target name? JorEl
        
        Starting with Bart
            1 away: [Lisa, Matthew]
            2 away: [Marge, Samantha]
            3 away: [Tyler]
            4 away: [Kyle]
            5 away: [Lex, Zod]
            6 away: [JorEl]
        found at a distance of 6

I asked people what kind of structure would be useful for keeping track of this kind of data and someone said a map. But what kind of map? Someone suggested that it would be good to keep track of the neighbors for each person. The neighbors are the friends. For example, Ashley's friends are Christopher, Emily, Jessica, and Joshua.

If we want a structure that keeps track of these kind of friendships, then we want to use names as keys into the structure. We ask the structure, "Who are the friends of Samantha?" or "Who are the friends of Ashley?". So a name, a String, will be used as the key. But what should it return? If we map a String to a String, then we can store only one friendship. We want to be able to return more than one friendship. Someone suggested that we want to use a set. That is exactly right.

The idea is that we want to have a map that converts a String into a Set of String values. Given the name of a person, we can get a Set with the names of that person's friends. For our sample file:

        "Andrew"        => maps to => [Christopher, Sarah]
        "Andrew"	=> maps to => [Christopher, Sarah]
        "Ashley"	=> maps to => [Christopher, Emily, Jessica, Joshua]
        "Bart"	        => maps to => [Lisa, Matthew]
        "Christopher"	=> maps to => [Andrew, Ashley, Jacob, Michael, Sarah]
        "Emily"	        => maps to => [Ashley, Joshua, Sarah]
        "Jacob"	        => maps to => [Christopher, Stuart]
        "Jessica"	=> maps to => [Ashley, Michael]
        "JorEl"	        => maps to => [KalEl, Zod]
        "Joshua"	=> maps to => [Ashley, Emily, Michael]
        "KalEl"	        => maps to => [JorEl]
        "Kyle"	        => maps to => [Lex, Tyler, Zod]
        "Lex"	        => maps to => [Kyle]
        "Lisa"	        => maps to => [Bart, Marge, Matthew]
        "Marge"	        => maps to => [Lisa]
        "Matthew"	=> maps to => [Bart, Lisa, Samantha]
        "Michael"	=> maps to => [Christopher, Jessica, Joshua]
        "Samantha"	=> maps to => [Matthew, Tyler]
        "Sarah"	        => maps to => [Andrew, Christopher, Emily]
        "Stuart"	=> maps to => [Jacob]
        "Tyler"	        => maps to => [Kyle, Samantha]
        "Zod"	        => maps to => [JorEl, Kyle]

Our first challenge, then, is to write code to construct such a structure. If it maps a String to a Set<String>, then it would be of this type:

        Map<String, Set<String>>

To construct one, we have to ask for a new TreeSet of this type:

        Map<String, Set<String>> friends = new TreeMap<String, Set<String>>();

That is a rather complex line of code, but the main complexity comes from what we are putting inside the "<" and ">" characters.

To fill up this structure, we need to process the input file. Remember that the input file has lines that have two names separated by a "--", as in:

    Ashley -- Christopher

We wrote the following code to read lines of input and find the ones that contain names:

        while (input.hasNextLine()) {
            String line = input.nextLine();
            if (line.contains("--")) {
                Scanner lineData = new Scanner(line);
                String name1 = lineData.next();
                lineData.next();  // this skips the "--" token
                String name2 = lineData.next();
                // process name1 and name2
            }
        }

This was not the interesting part of the code because we saw file processing in cse142. The interesting part is to think of how to process the two names. How do we update our friends map given a new friendship? Friendships are bidirectional, so we have to be careful to add the friendship in both directions. If there is an Ashley--Christopher friendship, then we have to make sure that Ashley's set of friends includes Christopher and we have to make sure that Christopher's set of friends includes Ashley.

At that point we ran out of time, so I said we would continue this in the next lecture.

Stuart Reges

Last modified: Wed Apr 18 12:50:28 PDT 2012