Packages  This Package  Prev  Next  Index  

§2.22 Class StreamTokenizer

public  class  java.io.StreamTokenizer
    extends  java.lang.Object  (I-§1.12)
{
        // Fields
    public double nval;	§2.22.1
    public String sval;	§2.22.2
    public int ttype;	§2.22.3

        // possible values for the ttype field
    public final static int TT_EOF;	§2.22.4
    public final static int TT_EOL;	§2.22.5
    public final static int TT_NUMBER;	§2.22.6
    public final static int TT_WORD;	§2.22.7

        // Constructors
    public StreamTokenizer(InputStream  I);	§2.22.8

        // Methods
    public void commentChar(int  ch);	§2.22.9
    public void eolIsSignificant(boolean  flag);	§2.22.10
    public int lineno();	§2.22.11
    public void lowerCaseMode(boolean  fl);	§2.22.12
    public int nextToken();	§2.22.13
    public void ordinaryChar(int  ch);	§2.22.14
    public void ordinaryChars(int  low, int  hi);	§2.22.15
    public void parseNumbers();	§2.22.16
    public void pushBack();	§2.22.17
    public void quoteChar(int  ch);	§2.22.18
    public void resetSyntax();	§2.22.19
    public void whitespaceChars(int  low, int  hi);	§2.22.20
    public void slashStarComments(boolean  flag);	§2.22.21
    public String toString();	§2.22.22
    public void whitespaceChars(int  low, int  hi);	§2.22.23
    public void wordChars(int  low, int  hi);	§2.22.24
}
The StreamTokenizer class takes an input stream and parses it into "tokens", allowing the tokens to be read one at a time. The parsing process is controlled by a table and a number of flags that can be set to various states. The stream tokenizer can recognize identifiers, numbers, quoted strings, and various comment styles.

Each byte read from the input stream is regarded as a character in the range '\u0000' through '\u00FF'. The character value is used to look up five possible attributes of the character: whitespace, alphabetic, numeric, string quote, and comment character. Each character can have zero or more of these attributes.

In addition an instance has four flags. These flags indicate:

A typical application first constructs an instance of this class, sets up the syntax tables, and then repeatedly loops calling the nextToken method (I-§2.22.13) in each iteration of the loop until it returns the value TT_EOF (I-§2.22.4).


Fields

nval

public double nval
If the current token is a number, this field contains the value of that number. The current token is a number when the value of the ttype field (I-§2.22.3) is TT_NUMBER (I-§2.22.6).

sval

public String sval
If the current token is a word token, this field contains a string giving the characters of the word token. When the current token is a quoted string token, this field contains the body of the string.
The current token is a word when when the value of the ttype field (I-§2.22.3) is TT_WORD (I-§2.22.7). The current token is a quoted string token when the value of the ttype field is a quote character (see I-§2.22.18).

ttype

public int ttype
After a call to the nextToken method (I-§2.22.13), this field contains the type of the token just read. For a single character token, its value is the single character, converted to an integer. For a quoted string token (see I-§2.22.18), its value is the quote character. Otherwise, its value is one of the following:

TT_EOF

public final static int TT_EOF = -1
A constant indicating that the end of the stream has been read.

TT_EOL

public final static int TT_EOL = '\n'
A constant indicating that the end of line has been read.

TT_NUMBER

public final static int TT_NUMBER = -2
A constant indicating that a number has been read.

TT_WORD

public final static int TT_WORD = -3
A constant indicating that a token has been read.

Constructors

StreamTokenizer

public StreamTokenizer(InputStream I)
Creates a stream tokenizer that parses the specified input stream. The stream tokenizer is initialized to the following default state:
Parameters:
I - the input stream

Methods

commentChar

public void commentChar(int ch)
Specified that the character argument starts a single line comment. All characters from the comment character to the end of the line are ignored by this stream tokenizer.
Parameters:
ch - the character

eolIsSignificant

public void eolIsSignificant(boolean flag)
If the flag argument is true, this tokenizer treats end of lines as tokens; the nextToken method (I-§2.22.13) returns TT_EOL (I-§2.22.5) and also sets the ttype field (I-§2.22.3) to this value when an end of line is read.


A line is a sequence of characters ending with either a carriage return character ('\r') or a newline character ('\n'). In addition, a carriage return character followed immediately by a newline character is treated as a single end- of-line token.
If the flag is false, end of line characters are treated as whitespace and serve only to separate tokens.
Parameters:
flag - true indicates that end-of-line characters are separate tokens; false indicates that end-of-line characters are whitespace

lineno

public int lineno()
Returns:
the current line number of this stream tokenizer

lowerCaseMode

public void lowerCaseMode(boolean fl)
If the flag argument is true, then the value in the sval field is lowercased whenever a word token is returned (the ttype field (I-§2.22.3) has the value TT_WORD (I-§2.22.7)) by the nextToken method (I-§2.22.13) of this tokenizer.


If the flag argument is false, then the sval field is not modified.
Parameters:
fl - true indicates that all word tokens should be lowercased

nextToken

public int nextToken() throws IOException
Parses the next token from the input stream of this tokenizer. The type of the next token is returned in the ttype field (I-§2.22.3). Additional information about the token may be in the nval field (I-§2.22.1) or the sval field (I-§2.22.2) of this tokenizer.

Returns:
The value of the ttype field (I-§2.22.3).
Throws
IOException (I-§2.29)
If an I/O error occurs.

ordinaryChar

public void ordinaryChar(int ch)
Specifies that the character argument is "ordinary" in this tokenizer. It removes any special significance the character has as a comment character, word component, string delimiter, whitespace, or number character. When such a character is encountered by the parser, the parser treates it as a single-character token and sets ttype field (I-§2.22.3) to the character value.

Parameters:
ch - the character

ordinaryChars

public void ordinaryChars(int low, int hi)
Specifies that all characters c in the range are "ordinary" in this tokenizer. See the ordinaryChar method (I-§2.22.14) for more information on a character being ordinary.

Parameters:
low - the low end of the range
hi - the high end of the range

parseNumbers

public void parseNumbers()
Specifies that numbers should be parsed by this tokenizer. The syntax table of this tokenizer is modified so that each of the twelve characters
has the "numeric" attribute.
When the parser encounters a word token that has the format of a double precision floating point number, it treates the token as a number rather than a word, by setting the the ttype field (I-§2.22.3) to the value TT_NUMBER (I-§2.22.6) and putting the numeric value of the token into the nval field (I-§2.22.1).

pushBack

public void pushBack()
Causes the next call to the nextToken method (I-§2.22.13) of this tokenizer to return the current value in the ttype field (I-§2.22.3), and not to modify the value in the nval (I-§2.22.1) or sval (I-§2.22.2) field.

quoteChar

public void quoteChar(int ch)
Specifies that matching pairs of this character delimit string constants in this tokenizer.
When the nextToken method (I-§2.22.13) encounters a string constant, the ttype field (I-§2.22.3) field is set to the string delimiter and the sval field (I-§2.22.2) is set to the body of the string.


If a string quote character is encountered, then a string is recognized, consisting of all characters after (but not including) the string quote character, up to (but not including) the next occurrence of that same string quote character, or a line terminator, or end of file. The usual escape sequences such as "\n" and "\t" are recognized and converted to single characters as the string is parsed.
Parameters:
ch - the character

resetSyntax

public void resetSyntax()
Resets this tokenizer's syntax table so that all characters are "ordinary." See the ordinaryChar method (I-§2.22.14) for more information on a character being ordinary.

slashSlashComments

public void slashSlashComments(boolean flag)
If the flag argument is true, this stream tokenizer recognizes C++ style comments. Any occurrence of two consecutive slash characters ('/') is treated as the beginning of a comment that extends to the end of the line.
If the flag argument is false, then C++ style comments are not treated specially.
Parameters:
flag - true indicates to recognize and ignore C++ style comments

slashStarComments

public void slashStarComments(boolean flag)
If the flag argument is true, this stream tokenizer recognizes C style comments. All text between successive occurrences of /* and */ are discarded.
If the flag argument is false, then C style comments are not treated specially.
Parameters:
flag - true indicates to recognize and ignore C style comments

toString

public String toString()
Returns the string representation of the current stream token.
Returns:
a string representation of the token specified by the ttype (I-§2.22.3), nval (I-§2.22.1), and sval (I-§2.22.2) fields.
Overrides:
toString in class Object (I-§1.12.9).

whitespaceChars

public void whitespaceChars(int low, int hi)
Specifies that all characters c in the range are whitespace character. Whitepsace characters serve only to separate tokens in the input stream.
Parameters:
low - the low end of the range
hi - the high end of the range

wordChars

public void wordChars(int low, int hi)
Specifies that all characters c in the range are word constituents. A word token consists of a word constitutent followed by zero or more word constituents or number constituents.
Parameters:
low - the low end of the range
hi - the high end of the range

Packages  This Package  Prev  Next  Index
Java API Document (HTML generated by dkramer on April 22, 1996)
Copyright © 1996 Sun Microsystems, Inc. All rights reserved
Please send any comments or corrections to doug.kramer@sun.com