Package org.terrier.indexing.tokenisation

Class TokenStream

  • All Implemented Interfaces:
    java.util.Iterator<java.lang.String>
    public abstract class TokenStream
extends java.lang.Object
implements java.util.Iterator<java.lang.String>
    Represents a stream of tokens found by a tokeniser. It is of note that a TokenStream may return null for a next() method, even if hasNext() previously returned true.
    Since:
    3.5
    Author:
    Craig Macdonald

    • Constructor Summary

      Constructors 
      Constructor Description
      TokenStream()  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void remove()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface java.util.Iterator

        forEachRemaining, hasNext, next

    • Constructor Detail

      • TokenStream

        public TokenStream()

    • Method Detail

      • remove

        public void remove()
        Specified by:
        remove in interface java.util.Iterator<java.lang.String>