Package org.dlese.dpc.index

Class Stemmer

java.lang.Object
org.dlese.dpc.index.Stemmer
public class Stemmer extends Object
Stemmer implements the Porter stemming algorithm. The Stemmer class transforms a word or array of words into their morphological root form. For example, this algorithm converts the words 'ocean,' 'oceans' and 'oceanic' to the single root 'ocean'.

The static methods getStem(String term) and getStems(String[] terms) can be used to quickly convert a word or words to their root form. Example code:

import org.dlese.dpc.index.Stemmer;

...

String word = "oceanic";
String stem = Stemmer.getStem(word); // stem now equals 'ocean'

String string = "A group of words that need to be stemmed";
String[] words = string.split("\\s+"); // Split on white space
String[] stems = Stemmer.getStems(words);
for(int i = 0; i invalid input: '<' stems.length; i++){
   ... do something with the stems ...
}

For more information about the Porter stemming algorithm, see http://www.tartarus.org/~martin/PorterStemmer .

Author:
Martin Porter, Unknown contributors, John Weatherley

  • Constructor Summary

    Constructors
    Constructor
    Description
    Stemmer()
    Constructor for the Stemmer object

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    add(char ch)
    Add a character to the word being stemmed.
    void
    add(char[] w, int wLen)
    Adds wLen characters to the word being stemmed contained in a portion of a char[] array.
    char[]
    getResultBuffer()
    Returns a reference to a character buffer containing the results of the stemming process.
    int
    getResultLength()
    Returns the length of the word resulting from the stemming process.
    static final String
    getStem(String term)
    Gets the stem of the given english word.
    static final String[]
    getStems(String[] terms)
    Gets the stems of the given english words.
    static void
    main(String[] args)
    Test program for demonstrating the Stemmer.
    void
    stem()
    Stem the word placed into the Stemmer buffer through calls to add().
    static final String
    stemWordsInLuceneClause(String string)
    Stems each of the words in a given Lucene clause String, returning the same String with the word parts in stemmed form.
    static final String
    stemWordsInString(String string)
    Stems each of the words or tokens in a given String, returning a String of stemmed tokens with all other characters removed.
    String
    toString()
    After a word has been stemmed, it can be retrieved by toString(), or a reference to the internal buffer can be retrieved by getResultBuffer and getResultLength (which is generally more efficient.)

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • Stemmer

      public Stemmer()
      Constructor for the Stemmer object

  • Method Details

    • getStem

      public static final String getStem(String term)
      Gets the stem of the given english word. For proper results, the input should contain letters only [a-zA-Z].
      Parameters:
      term - A term in english.
      Returns:
      The stem value of the english term.

    • getStems

      public static final String[] getStems(String[] terms)
      Gets the stems of the given english words. For proper results, the input should contain letters only [a-zA-Z].
      Parameters:
      terms - A group of terms in english.
      Returns:
      The stems values for each term.

    • stemWordsInString

      public static final String stemWordsInString(String string)
      Stems each of the words or tokens in a given String, returning a String of stemmed tokens with all other characters removed. Token characters include letters and numbers [a-zA-Z0-9], representing the class of tokens that are searchable by Lucene. Note: the tokens "AND" and "OR" (upper case) are left unchanged.

      Example:

      oceans and rain AND 44rains http://dlese.org/oceans
          is transformed to
      ocean and rain AND 44rain http dlese org ocean

      Parameters:
      string - A word, phrase, or any arbitrary String.
      Returns:
      A String containing each letter/number token in stemmed form.

    • stemWordsInLuceneClause

      public static final String stemWordsInLuceneClause(String string)
      Stems each of the words in a given Lucene clause String, returning the same String with the word parts in stemmed form. The method leaves all words that may be found in a Lucene clause unchanged such as 'AND', 'OR' and field specificers such as 'titles:'.

      Example:

      titles:("oceans AND oceans44 OR 44oceans and oceanic")^20 or cooled
          is transformed to
      titles:("ocean AND oceans44 OR 44ocean and ocean")^20 or cool

      Parameters:
      string - A word, phrase, Lucene clause, or any arbitrary String.
      Returns:
      Each non-clause word is stemmed in place, leaving non-word characters and clause words unchanged.

    • add

      public void add(char ch)
      Add a character to the word being stemmed. When you are finished adding characters, you can call stem(void) to stem the word.
      Parameters:
      ch - DESCRIPTION

    • add

      public void add(char[] w, int wLen)
      Adds wLen characters to the word being stemmed contained in a portion of a char[] array. This is like repeated calls of add(char ch), but faster.
      Parameters:
      w - DESCRIPTION
      wLen - DESCRIPTION

    • toString

      public String toString()
      After a word has been stemmed, it can be retrieved by toString(), or a reference to the internal buffer can be retrieved by getResultBuffer and getResultLength (which is generally more efficient.)
      Overrides:
      toString in class Object
      Returns:
      DESCRIPTION

    • getResultLength

      public int getResultLength()
      Returns the length of the word resulting from the stemming process.
      Returns:
      The resultLength value

    • getResultBuffer

      public char[] getResultBuffer()
      Returns a reference to a character buffer containing the results of the stemming process. You also need to consult getResultLength() to determine the length of the result.
      Returns:
      The resultBuffer value

    • stem

      public void stem()
      Stem the word placed into the Stemmer buffer through calls to add(). Returns true if the stemming process resulted in a word different from the input. You can retrieve the result with getResultLength()/getResultBuffer() or toString().

    • main

      public static void main(String[] args)
      Test program for demonstrating the Stemmer. It reads text from a a list of files, stems each word, and writes the result to standard output. Note that the word stemmed is expected to be in lower case: forcing lower case must be done outside the Stemmer class. Usage: Stemmer file-name file-name ...
      Parameters:
      args - The command line arguments