org.exist.storage.analysis
Class SimpleTokenizer

java.lang.Object
  org.exist.storage.analysis.SimpleTokenizer

All Implemented Interfaces:

Tokenizer

public class SimpleTokenizer
extends Object
implements Tokenizer

This is the default class used by the fulltext indexer for tokenizing a string into words. Known token types are defined by class Token.

Author:

Wolfgang Meier

Constructor Summary

SimpleTokenizer()

SimpleTokenizer(boolean stem)

Method Summary

int	getLength()
String	getText()
static void	main(String[] args)
TextToken	nextToken()
TextToken	nextToken(boolean wildcards)
void	setStemming(boolean stem)
void	setText(CharSequence text)
void	setText(CharSequence text, int offset)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleTokenizer

public SimpleTokenizer()

SimpleTokenizer

public SimpleTokenizer(boolean stem)

Method Detail

setStemming

public void setStemming(boolean stem)

Specified by:

setStemming in interface Tokenizer

getLength

public int getLength()

getText

public String getText()

nextToken

public TextToken nextToken()

Specified by:

nextToken in interface Tokenizer

nextToken

public TextToken nextToken(boolean wildcards)

Specified by:

nextToken in interface Tokenizer

setText

public void setText(CharSequence text)

Specified by:

setText in interface Tokenizer

setText

public void setText(CharSequence text,
                    int offset)

Specified by:

setText in interface Tokenizer

main

public static void main(String[] args)