|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object | +--dm.dmql.RawTextTokenStream
Tokenizes a generic text document
Field Summary | |
protected boolean |
_bHoldCommaForNext
|
protected java.io.BufferedReader |
_brInput
Non-static Data Members |
protected int |
_nLine
|
protected int |
_nLinePos
|
protected int |
_nPos
|
protected Token |
_tLookAhead
|
protected static int |
EOF
Static Constants |
Constructor Summary | |
RawTextTokenStream()
Constructor - default |
Method Summary | |
void |
close()
Close the file currently being tokenized |
void |
match(java.lang.String content)
Attempts to match the next token, throws TokenStreamException |
Token |
nextToken()
Returns the next Token from a TokenStream or null if the end of the stream has been reached. |
boolean |
nextTokenEquals(java.lang.String str)
Determines if the next token is the specified String |
Token |
nextTokenInternal()
Returns the next Token from a TokenStream or null if the end of the stream has been reached. |
void |
process(java.lang.String str)
Process a String for tokenizing |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
protected static final int EOF
protected java.io.BufferedReader _brInput
protected int _nLine
protected int _nLinePos
protected int _nPos
protected boolean _bHoldCommaForNext
protected Token _tLookAhead
Constructor Detail |
public RawTextTokenStream()
Method Detail |
public void process(java.lang.String str) throws TokenStreamException
process
in interface TokenStream
str
- String for tokenizingpublic void close() throws TokenStreamException
close
in interface TokenStream
public void match(java.lang.String content) throws TokenStreamException
match
in interface TokenStream
content
- String to match next token againstpublic boolean nextTokenEquals(java.lang.String str) throws TokenStreamException
nextTokenEquals
in interface TokenStream
str
- The String to be compared againstpublic Token nextToken() throws TokenStreamException
nextToken
in interface TokenStream
public Token nextTokenInternal() throws TokenStreamException
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |