dm.dmql
Interface TokenStream

All Known Implementing Classes:
RawTextTokenStream

public interface TokenStream

A generic tokenizer interface

Author:
Scott Sanner

Method Summary
 void close()
          Closes the TokenStream
 void match(java.lang.String content)
          Attempts to match the next token, throws TokenStreamException
 Token nextToken()
          Returns the next Token from a TokenStream or null if the end of the stream has been reached.
 boolean nextTokenEquals(java.lang.String str)
          Determines if the next token is the specified String
 void process(java.lang.String str)
          Opens the TokenStream given a string to process
 

Method Detail

process

public void process(java.lang.String str)
             throws TokenStreamException
Opens the TokenStream given a string to process
Parameters:
str - The String to process

close

public void close()
           throws TokenStreamException
Closes the TokenStream

match

public void match(java.lang.String content)
           throws TokenStreamException
Attempts to match the next token, throws TokenStreamException
Parameters:
content - String to match next token against

nextTokenEquals

public boolean nextTokenEquals(java.lang.String str)
                        throws TokenStreamException
Determines if the next token is the specified String
Parameters:
str - The String to be compared against
Returns:
true if next token is equal to str

nextToken

public Token nextToken()
                throws TokenStreamException
Returns the next Token from a TokenStream or null if the end of the stream has been reached. Throws a TokenStreamException if there is an IO error.
Returns:
The next token or null (if no tokens remain)