dm.dmql
Class RawTextTokenStream

java.lang.Object
  |
  +--dm.dmql.RawTextTokenStream
All Implemented Interfaces:
TokenStream

public class RawTextTokenStream
extends java.lang.Object
implements TokenStream

Tokenizes a generic text document

Author:
Scott Sanner

Field Summary
protected  boolean _bHoldCommaForNext
           
protected  java.io.BufferedReader _brInput
          Non-static Data Members
protected  int _nLine
           
protected  int _nLinePos
           
protected  int _nPos
           
protected  Token _tLookAhead
           
protected static int EOF
          Static Constants
 
Constructor Summary
RawTextTokenStream()
          Constructor - default
 
Method Summary
 void close()
          Close the file currently being tokenized
 void match(java.lang.String content)
          Attempts to match the next token, throws TokenStreamException
 Token nextToken()
          Returns the next Token from a TokenStream or null if the end of the stream has been reached.
 boolean nextTokenEquals(java.lang.String str)
          Determines if the next token is the specified String
 Token nextTokenInternal()
          Returns the next Token from a TokenStream or null if the end of the stream has been reached.
 void process(java.lang.String str)
          Process a String for tokenizing
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

EOF

protected static final int EOF
Static Constants

_brInput

protected java.io.BufferedReader _brInput
Non-static Data Members

_nLine

protected int _nLine

_nLinePos

protected int _nLinePos

_nPos

protected int _nPos

_bHoldCommaForNext

protected boolean _bHoldCommaForNext

_tLookAhead

protected Token _tLookAhead
Constructor Detail

RawTextTokenStream

public RawTextTokenStream()
Constructor - default
Method Detail

process

public void process(java.lang.String str)
             throws TokenStreamException
Process a String for tokenizing
Specified by:
process in interface TokenStream
Parameters:
str - String for tokenizing

close

public void close()
           throws TokenStreamException
Close the file currently being tokenized
Specified by:
close in interface TokenStream

match

public void match(java.lang.String content)
           throws TokenStreamException
Attempts to match the next token, throws TokenStreamException
Specified by:
match in interface TokenStream
Parameters:
content - String to match next token against

nextTokenEquals

public boolean nextTokenEquals(java.lang.String str)
                        throws TokenStreamException
Determines if the next token is the specified String
Specified by:
nextTokenEquals in interface TokenStream
Parameters:
str - The String to be compared against
Returns:
true if next token is equal to str

nextToken

public Token nextToken()
                throws TokenStreamException
Returns the next Token from a TokenStream or null if the end of the stream has been reached. Throws a TokenStreamException if there is an IO error.
Specified by:
nextToken in interface TokenStream
Returns:
The next token or null (if no tokens remain)

nextTokenInternal

public Token nextTokenInternal()
                        throws TokenStreamException
Returns the next Token from a TokenStream or null if the end of the stream has been reached. Throws a TokenStreamException if there is an IO error.
Returns:
The next token or null (if no tokens remain)