CSE 331: Batch processing

lecture

Course motto: Your job is to deliver code that you know works.

This lecture also returns to Lecture 2's last sections, Stronger and weaker through Knowing a program in pieces. Since these notes do not repeat them, read them in Lecture 2's notes.

Sum, from the command line

Lectures 1 and 2 studied a Sum that asks for numbers, prints them, and prints their sum. The program below is a different Sum. It adds up numbers too, but it is 99 lines long, and most of its lines have nothing to do with adding. Read the whole program before you read on, with three questions in mind. What does Sum do? What is good about it? What is not? If some of its Java is new to you, such as the try on line 59, read past it. These notes come back to each part that matters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

/**
 * Reads whitespace-separated integers and prints either their sum
 * (default) or their average (with -a/--average).
 *
 * Usage:
 *   java Sum [options] [file]
 *
 * Options:
 *   -a, --average    print the average instead of the sum
 *   --precision=N    digits after the decimal point (only valid
 *                    with -a; defaults to full precision)
 *
 * If a file argument is given, input is read from that file;
 * otherwise it is read from standard input.
 */
public class Sum {

    public static void main(String[] args) {
        boolean average = false;
        boolean precisionGiven = false;
        int precision = 0;
        String filename = null;

        for (String arg : args) {
            if (arg.equals("-a") || arg.equals("--average")) {
                average = true;
            } else if (arg.startsWith("--precision=")) {
                precisionGiven = true;
                String value = arg.substring("--precision=".length());
                try {
                    precision = Integer.parseInt(value);
                } catch (NumberFormatException e) {
                    fail("--precision must be an integer, got: "
                            + value);
                }
                if (precision < 0) {
                    fail("--precision must not be negative, got: "
                            + precision);
                }
            } else if (arg.startsWith("-")) {
                fail("unknown option: " + arg);
            } else if (filename == null) {
                filename = arg;
            } else {
                fail("too many file arguments: " + arg);
            }
        }

        if (precisionGiven && !average) {
            fail("--precision is only allowed with -a/--average");
        }

        long sum = 0;
        long count = 0;
        try (Scanner scanner = openInput(filename)) {
            while (scanner.hasNext()) {
                if (scanner.hasNextInt()) {
                    sum += scanner.nextInt();
                    count++;
                } else {
                    fail("not an integer: " + scanner.next());
                }
            }
        } catch (FileNotFoundException e) {
            fail("could not read input: " + e.getMessage());
        }

        if (average) {
            if (count == 0) {
                fail("cannot average zero values");
            }
            double avg = (double) sum / count;
            if (precisionGiven) {
                System.out.printf("%." + precision + "f%n", avg);
            } else {
                System.out.println(avg);
            }
        } else {
            System.out.println(sum);
        }
    }

    private static Scanner openInput(String filename)
            throws FileNotFoundException {
        if (filename == null) {
            return new Scanner(System.in);
        }
        return new Scanner(new File(filename));
    }

    private static void fail(String message) {
        System.err.println("Sum: " + message);
        System.exit(1);
    }
}

Compile and run Sum as you did Lecture 1's Sum. With nothing after java Sum, it reads numbers from the keyboard until you end the input. Each run in these notes shows the command, then anything typed, then what Sum printed:

java Sum
3 4 5
12

With -a, Sum prints the average instead:

java Sum -a
3 4 5
4.0

Given the name of a file, Sum reads its numbers from that file. Here the file nums.txt contains 3 4 5:

java Sum nums.txt
12

So far this Sum looks like Lecture 1's without the prompt and the Got line. The runs that broke Lecture 1's "user manual" sentence tell the two apart. On 1 2 three 4, Lecture 1's Sum added 1 and 2 and ignored the rest. On 2147483647 1, it printed a negative number. Before reading on, predict what this Sum does with each:

java Sum
1 2 three 4
Sum: not an integer: three

java Sum
2147483647 1
2147483648

This Sum stops at the word, and its message says which word it could not use. It also gets the second sum right. Line 57 declares sum as a long, Java's 64-bit type for whole numbers, so this addition does not wrap.

From here on, Sum means this lecture's Sum.

Batch programs

Lecture 1's Sum and this one are both programs of one kind:

What is Sum's input? The numbers are only part of the answer. The words typed after java Sum are input too. The user chooses them, and they change what Sum does. These words are Sum's command-line arguments. Java hands them to Sum's main as the array args, in the order they were typed. The loop on line 28 reads them one at a time. An argument that starts with a dash, such as -a or --precision=2, is an option. Sum takes any other argument as the name of a file, and reads the numbers from that file. With no file named, Sum reads the numbers from standard input, which by default is the keyboard. Lines 89 and 90 make that choice.

The run on 1 2 three 4 printed a message instead of a sum. The message is output too, but Sum keeps it apart from its results. Results go to standard output (lines 78 to 83), and messages go to standard error (line 96, in fail). When Sum exits, it also leaves behind an exit status, a number that says whether the run succeeded. By convention, 0 means success and anything else means failure. fail exits with status 1 (line 97). Java sets a status even for a program that never chooses one: 0 if main finishes, and 1 if the program ends in a stack trace. In a terminal, though, results and messages look the same, and the exit status does not show at all.

Why keep results and messages apart, if they look the same? Send standard output to a file with >, and the difference shows. The file bad.txt contains 3 x. First predict what each of these two runs shows on the terminal and leaves in total.txt:

java Sum nums.txt > total.txt

java Sum bad.txt > total.txt
Sum: not an integer: x

The first run shows nothing, and leaves 12 in total.txt. The second still shows its message, since > sends only standard output to the file. It leaves total.txt empty. Either way, the file never contains a message where a sum should be.

The exit status says which run failed. On macOS and Linux, echo $? prints the status of the last command: 1 right after the run on bad.txt, and 0 right after the run on nums.txt. In Windows PowerShell, $LASTEXITCODE contains the status instead.

Both macOS and Linux are modeled on an older operating system, Unix. Unix did not always keep results and messages apart. At first, every Unix program printed its messages on standard output. That caused trouble when the output went to a file. It caused worse trouble once pipes let one program's output flow straight into another program as input. At Bell Labs, where Unix was built, a program's output could go to the lab's typesetter, the machine that printed its papers. A program's messages went there too. Once, several people typesetting documents got back "a single, beautifully typeset line": cannot open file foobar.¹ Soon after, Dennis Ritchie, one of Unix's two creators, added standard error. Standard error alone was not quite enough. With several programs in one pipeline, a message also had to say which program wrote it. Sum follows that convention: each of its messages begins with Sum:.

In all, what a user sees of Sum has three parts: its standard output, its standard error, and its exit status. Sum's specification has to say what each one is, on every run. A person at the terminal reads the first two. A script, a program that runs other programs, can read all three. A script that runs Sum cannot make sense of a message where it expected a number. Such a script is like a client in Lecture 2. It relies on what Sum promises, and it is correct if it works with every version of Sum that keeps those promises. Unlike a method's client, though, the script owes Sum nothing. Like any user, it may give Sum anything at all.

Expecting other programs to read a program's output is an old idea. A 1978 account of the Unix style lists four maxims. The second begins: "Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information."² A message among Sum's results would be that clutter, and Sum keeps it out by writing its messages on standard error. The words "as yet unknown" matter too. Sum's author cannot know which programs will read Sum's output, so whatever those programs rely on has to be in Sum's specification.

Most programs people use are not batch programs. A web browser does not read its input, compute, and exit. It keeps running, and responds to whatever the user does next, until the user tells it to stop. A later lecture takes up programs of that kind.

The user manual, on every run

Sum's specification, then, has to say what its standard output, its standard error, and its exit status are on every run. Sum already comes with a user manual: the comment on lines 5 to 19. It speaks to whoever runs Sum, not to whoever reads its code. Lecture 1 called a whole program's specification a precise version of its user manual. Some programmers even write the manual first, as "a precise specification of the job to be done."³ Let's see how close this manual comes to being a specification, by testing it as Lecture 1 tested its first "user manual" sentence: against runs its author may not have pictured.

The manual promises to read integers, and 2147483648 is an integer:

java Sum
2147483648
Sum: not an integer: 2147483648

Sum reads only the integers that fit in an int, since line 61 asks hasNextInt. Lecture 1's specification needed its range clause for the same reason. Next, type nothing and end the input. Sum prints 0, an answer the manual covers if the sum of no numbers is 0, as Lecture 1 granted. With -a, though, the same empty input gets a message instead of an average:

java Sum -a
Sum: cannot average zero values

The manual says nothing about this run, nor about the run on 1 2 three 4, nor about the exit status of any run.

Lecture 1 set a standard for every program in this course: every input a user can produce gets a response the specification describes, and a stack trace is never that response. So far, Sum passes the second part: no run of Sum above ends in a stack trace. Each error in them goes through fail, on lines 95 to 98, which prints on standard error a message that starts with the program's name, and then exits with status 1. That consistency is part of what is good about Sum: every error above looks the same, to a person and to a script.

Sum fails the first part. The manual describes none of Sum's error responses, neither the messages nor the exit status. Those are responses its user sees, so a specification has to describe them. The manual also sits where its user never sees it, in the source code. Asking for it with java Sum --help gets only Sum: unknown option: --help. Exercises 1 and 5 ask you to fix both.

Checks at the edge

Sum does a lot of checking. Lines 28 to 55 check its arguments, line 61 checks each word of its input, and lines 73 to 75 check that there is something to average. What are those checks for? Take one away and see. Below are two changed copies of Sum, each with one check removed and its lines left blank, so that every other line keeps its number.

The first copy lacks lines 40 to 43, which check that the precision is not negative. With the check, Sum answers a negative precision with a message at once, before it reads any input:

java Sum -a --precision=-1
Sum: --precision must not be negative, got: -1

What does the copy without the check do with the same command, given the numbers 1 2?

java Sum -a --precision=-1
1 2
Exception in thread "main" java.util.UnknownFormatConversionException: Conversion = '.'
    at java.base/java.util.Formatter.parse(Formatter.java:2818)
    at java.base/java.util.Formatter.format(Formatter.java:2744)
    at java.base/java.io.PrintStream.format(PrintStream.java:1183)
    at java.base/java.io.PrintStream.printf(PrintStream.java:1081)
    at Sum.main(Sum.java:78)

The negative precision reached the printf on line 78, which builds its format from the precision. A precision of 2 gives the format %.2f, which prints two digits after the decimal point. A precision of -1 gives %.-1f, which is not a format printf accepts. So the call throws, and the user gets a stack trace. With the check on lines 40 to 43 in place, a negative precision never reaches that call.

The second copy lacks lines 73 to 75, which check that some number was read. On empty input with -a, it does not crash:

java Sum -a
NaN

NaN, short for "not a number," is what the division on line 76 gives when it divides 0 by 0. Worse, the exit status is 0, so the run reports success. A script that runs java Sum -a and checks the status would take NaN for an average. A crash would at least have said that something went wrong.

What should the average of no numbers be? Sum refuses to give one, and prints a message. Printing 0 would be just as reasonable. There is no single right answer, but "the one guaranteed wrong answer is to ignore the problem."⁴ The copy without lines 73 to 75 does just that. A program that did the same once stopped a Navy cruiser, the USS Yorktown, at sea for a couple of hours, after a crew member entered a zero and the program divided by it. What matters is that the manual says which answer Sum gives, and so far it says neither.

So each check stands at what Lecture 1 called the edge of the program, between the user and the computation. The user owes Sum nothing, so the checks take whatever arrives and pass on only what the computation can handle: a precision that is not negative, and a count that is not zero.⁵ That suggests a way to organize a program like Sum. Keep the code that deals with the user (the arguments, the messages, the exit status) apart from the code that computes the answer. Sum does nearly all its work in main, both kinds together. For a program of 99 lines, keeping the work there and splitting it up can both be defended.

Split it, and Lecture 2 says what the pieces should be: methods with specifications. The checks stay at the edge, in main. A method inside may then have a precondition, such as a count that is not zero, since its client is main, code that can be held to a promise. Lecture 2's exercise 6 made the same argument. Exercise 3 below asks you to do the split.

Homework 1's Task 4 has the same shape. Gradebook is a batch program too, with its user manual at the top of its source. Its main has to check the three files it reads before it calls computeCourseGrades, whose @requires clauses assume those checks passed. Task 4 also asks for Lecture 1's standard: when the user makes a mistake, a message and an exit, never a stack trace.

Exercises

Each answer is folded under its question. Write yours first.

1. Repair Sum's user manual until it is true of every run of Sum in these notes, error messages and exit status included.

2. Find an input whose error message misleads the user.

3. Split Sum as Checks at the edge suggests. Leave in main the code that deals with the user: the arguments, the input, the messages, and the exit status. Move into a method the code that computes what Sum prints from the numbers read, and write that method's specification. Which of Sum's checks does the method's precondition rely on, and why may it rely on them?

4. Here are two manuals for what Sum does on a word that is not a whole number:

• M1. It prints Sum: not an integer: and the word on standard error, and exits with status 1.
• M2. It exits with a nonzero status.

Which is stronger? A script runs Sum and checks only whether Sum succeeded. Against which of M1 and M2 is the script correct?

5. Sum's user manual is a comment in its source, where someone running Sum never sees it. Asking for help gets only an error:

java Sum --help
Sum: unknown option: --help

Give Sum a --help option that prints the manual. When the arguments contain a mistake, such as an unknown option, make Sum say how to get the manual. Then add to the manual what each case prints, where, and with what exit status.

Further reading

• The maxims. M. D. McIlroy, E. N. Pinson, and B. A. Tague, "UNIX Time-Sharing System: Foreword" (The Bell System Technical Journal, 1978). In six pages, the foreword describes a style in which large tasks are built from small, loosely coupled parts, then gives the four maxims under the heading "Style," with an illustration of each.
• How the conventions arose. M. D. McIlroy, "A Research UNIX Reader" (Bell Labs Computing Science Technical Report 139, 1987). McIlroy, who was there, tells in short notes how the early Unix programs and their manual took shape. The note on pipes tells how messages moved off standard output and came to name their program. The copy online has the notes but not the manual pages they annotate.
• The Unix philosophy. Eric S. Raymond, The Art of Unix Programming (2003). Chapter 1 quotes the four maxims, then sets out seventeen rules of the Unix tradition. One is the Rule of Repair: "when you must fail, fail noisily and as soon as possible," since the worst bugs are those whose damage shows up only much later. The copy of Sum without its count check breaks this rule: it prints NaN and reports success.
• Command-line programs today. Aanand Prasad, Ben Firshman, Carl Tashian, and Eva Parish, Command Line Interface Guidelines, a guide that updates the Unix tradition and puts the person at the terminal first. Three of its basic rules are this lecture's: exit status 0 on success and nonzero on failure, results on standard output, and messages on standard error. Later sections advise printing help on --help, as in exercise 5, and checking input early with understandable errors, as in Checks at the edge.