Package org.apache.lucene.analysis

Class Analyzer

  • All Implemented Interfaces:
    Closeable, AutoCloseable
    Direct Known Subclasses:
    AnalyzerWrapper
    public abstract class Analyzer
extends Object
implements Closeable
    An Analyzer builds TokenStreams, which analyze text. It thus represents a policy for extracting index terms from text.

    In order to define what analysis is done, subclasses must define their TokenStreamComponents in createComponents(String, Reader). The components are then reused in each call to tokenStream(String, Reader).

    Simple example: 

     Analyzer analyzer = new Analyzer() {
  @Override
   protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
     Tokenizer source = new FooTokenizer(reader);
     TokenStream filter = new FooFilter(source);
     filter = new BarFilter(filter);
     return new TokenStreamComponents(source, filter);
   }
 };
    For more examples, see the Analysis package documentation.

    For some concrete implementations bundled with Lucene, look in the analysis modules:

    • Common: Analyzers for indexing content in different languages and domains.
    • ICU: Exposes functionality from ICU to Apache Lucene.
    • Kuromoji: Morphological analyzer for Japanese text.
    • Morfologik: Dictionary-driven lemmatization for the Polish language.
    • Phonetic: Analysis for indexing phonetic signatures (for sounds-alike search).
    • Smart Chinese: Analyzer for Simplified Chinese, which indexes words.
    • Stempel: Algorithmic Stemmer for the Polish Language.
    • UIMA: Analysis integration with Apache UIMA.

    • Method Detail

      • initReader

        protected Reader initReader​(String fieldName,
                            Reader reader)
        Override this if you want to add a CharFilter chain.

        The default implementation returns reader unchanged.

        Parameters:
        fieldName - IndexableField name being indexed
        reader - original Reader
        Returns:
        reader, optionally decorated with CharFilter(s)

      • getPositionIncrementGap

        public int getPositionIncrementGap​(String fieldName)
        Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.
        Parameters:
        fieldName - IndexableField name being indexed.
        Returns:
        position increment gap, added to the next token emitted from tokenStream(String,Reader). This value must be >= 0.

      • getOffsetGap

        public int getOffsetGap​(String fieldName)
        Just like getPositionIncrementGap(java.lang.String), except for Token offsets instead. By default this returns 1. This method is only called if the field produced at least one token for indexing.
        Parameters:
        fieldName - the field just indexed
        Returns:
        offset gap, added to the next token emitted from tokenStream(String,Reader). This value must be >= 0.

      • close

        public void close()
        Frees persistent resources used by this Analyzer
        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface Closeable