Package org.apache.lucene.search.similarities

Class BM25Similarity

java.lang.Object
org.apache.lucene.search.similarities.Similarity
org.apache.lucene.search.similarities.BM25Similarity
public class BM25Similarity extends Similarity
BM25 Similarity. Introduced in Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 1994). Gaithersburg, USA, November 1994.

  • Field Details

    • discountOverlaps

      protected boolean discountOverlaps
      True if overlap tokens (tokens with a position of increment of zero) are discounted from the document's length.

  • Constructor Details

    • BM25Similarity

      public BM25Similarity(float k1, float b)
      BM25 with the supplied parameter values.
      Parameters:
      k1 - Controls non-linear term frequency normalization (saturation).
      b - Controls to what degree document length normalizes tf values.

    • BM25Similarity

      public BM25Similarity()
      BM25 with these default values:
      • k1 = 1.2,
      • b = 0.75.

  • Method Details

    • idf

      protected float idf(long docFreq, long numDocs)
      Implemented as log(1 + (numDocs - docFreq + 0.5)/(docFreq + 0.5)).

    • sloppyFreq

      protected float sloppyFreq(int distance)
      Implemented as 1 / (distance + 1).

    • scorePayload

      protected float scorePayload(int doc, int start, int end, BytesRef payload)
      The default implementation returns 1

    • avgFieldLength

      protected float avgFieldLength(CollectionStatistics collectionStats)
      The default implementation computes the average as sumTotalTermFreq / maxDoc, or returns 1 if the index does not store sumTotalTermFreq (Lucene 3.x indexes or any field that omits frequency information).

    • encodeNormValue

      protected byte encodeNormValue(float boost, int fieldLength)
      The default implementation encodes boost / sqrt(length) with SmallFloat.floatToByte315(float). This is compatible with Lucene's default implementation. If you change this, then you should change decodeNormValue(byte) to match.

    • decodeNormValue

      protected float decodeNormValue(byte b)
      The default implementation returns 1 / f2 where f is SmallFloat.byte315ToFloat(byte).

    • setDiscountOverlaps

      public void setDiscountOverlaps(boolean v)
      Sets whether overlap tokens (Tokens with 0 position increment) are ignored when computing norm. By default this is true, meaning overlap tokens do not count when computing norms.

    • getDiscountOverlaps

      public boolean getDiscountOverlaps()
      Returns true if overlap tokens are discounted from the document's length.
      See Also:

    • computeNorm

      public final long computeNorm(FieldInvertState state)
      Description copied from class: Similarity
      Computes the normalization value for a field, given the accumulated state of term processing for this field (see FieldInvertState).

      Matches in longer fields are less precise, so implementations of this method usually set smaller values when state.getLength() is large, and larger values when state.getLength() is small.

      Specified by:
      computeNorm in class Similarity
      Parameters:
      state - current processing state for this field
      Returns:
      computed norm value

    • idfExplain

      public Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics termStats)
      Computes a score factor for a simple term and returns an explanation for that score factor.

      The default implementation uses: 

       idf(docFreq, searcher.maxDoc());
      Note that CollectionStatistics.maxDoc() is used instead of IndexReader#numDocs() because also TermStatistics.docFreq() is used, and when the latter is inaccurate, so is CollectionStatistics.maxDoc(), and in the same direction. In addition, CollectionStatistics.maxDoc() is more efficient to compute
      Parameters:
      collectionStats - collection-level statistics
      termStats - term-level statistics for the term
      Returns:
      an Explain object that includes both an idf score factor and an explanation for the term.

    • idfExplain

      public Explanation idfExplain(CollectionStatistics collectionStats, TermStatistics[] termStats)
      Computes a score factor for a phrase.

      The default implementation sums the idf factor for each term in the phrase.

      Parameters:
      collectionStats - collection-level statistics
      termStats - term-level statistics for the terms in the phrase
      Returns:
      an Explain object that includes both an idf score factor for the phrase and an explanation for each term.

    • computeWeight

      public final Similarity.SimWeight computeWeight(float queryBoost, CollectionStatistics collectionStats, TermStatistics... termStats)
      Description copied from class: Similarity
      Compute any collection-level weight (e.g. IDF, average document length, etc) needed for scoring a query.
      Specified by:
      computeWeight in class Similarity
      Parameters:
      queryBoost - the query-time boost.
      collectionStats - collection-level statistics, such as the number of tokens in the collection.
      termStats - term-level statistics, such as the document frequency of a term across the collection.
      Returns:
      SimWeight object with the information this Similarity needs to score a query.

    • simScorer

      public final Similarity.SimScorer simScorer(Similarity.SimWeight stats, AtomicReaderContext context) throws IOException
      Description copied from class: Similarity
      Creates a new Similarity.SimScorer to score matching documents from a segment of the inverted index.
      Specified by:
      simScorer in class Similarity
      Parameters:
      stats - collection information from Similarity.computeWeight(float, CollectionStatistics, TermStatistics...)
      context - segment of the inverted index to be scored.
      Returns:
      SloppySimScorer for scoring documents across context
      Throws:
      IOException - if there is a low-level I/O error

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getK1

      public float getK1()
      Returns the k1 parameter
      See Also:

    • getB

      public float getB()
      Returns the b parameter
      See Also: