Class NumericTokenStream
- All Implemented Interfaces:
Closeable
,AutoCloseable
TokenStream
for indexing numeric values that can be used by NumericRangeQuery
or NumericRangeFilter
.
Note that for simple usage, IntField
, LongField
, FloatField
or DoubleField
is
recommended. These fields disable norms and
term freqs, as they are not usually needed during
searching. If you need to change these settings, you
should use this class.
Here's an example usage, for an int
field:
FieldType fieldType = new FieldType(TextField.TYPE_NOT_STORED); fieldType.setOmitNorms(true); fieldType.setIndexOptions(IndexOptions.DOCS_ONLY); Field field = new Field(name, new NumericTokenStream(precisionStep).setIntValue(value), fieldType); document.add(field);
For optimal performance, re-use the TokenStream and Field instance for more than one document:
NumericTokenStream stream = new NumericTokenStream(precisionStep); FieldType fieldType = new FieldType(TextField.TYPE_NOT_STORED); fieldType.setOmitNorms(true); fieldType.setIndexOptions(IndexOptions.DOCS_ONLY); Field field = new Field(name, stream, fieldType); Document document = new Document(); document.add(field); for(all documents) { stream.setIntValue(value) writer.addDocument(document); }
This stream is not intended to be used in analyzers; it's more for iterating the different precisions during indexing a specific numeric value.
NOTE: as token streams are only consumed once
the document is added to the index, if you index more
than one numeric field, use a separate NumericTokenStream
instance for each.
See NumericRangeQuery
for more details on the
precisionStep
parameter as well as how numeric fields work under the hood.
- Since:
- 2.9
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic interface
Expert: Use this attribute to get the details of the currently generated token.static final class
Implementation ofNumericTokenStream.NumericTermAttribute
.Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionCreates a token stream for numeric values using the defaultprecisionStep
NumericUtils.PRECISION_STEP_DEFAULT
(4).NumericTokenStream
(int precisionStep) Creates a token stream for numeric values with the specifiedprecisionStep
.NumericTokenStream
(AttributeSource.AttributeFactory factory, int precisionStep) Expert: Creates a token stream for numeric values with the specifiedprecisionStep
using the givenAttributeSource.AttributeFactory
. -
Method Summary
Modifier and TypeMethodDescriptionint
Returns the precision step.boolean
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.setDoubleValue
(double value) Initializes the token stream with the supplieddouble
value.setFloatValue
(float value) Initializes the token stream with the suppliedfloat
value.setIntValue
(int value) Initializes the token stream with the suppliedint
value.setLongValue
(long value) Initializes the token stream with the suppliedlong
value.Methods inherited from class org.apache.lucene.analysis.TokenStream
close, end
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
Field Details
-
TOKEN_TYPE_FULL_PREC
The full precision token gets this token type assigned.- See Also:
-
TOKEN_TYPE_LOWER_PREC
The lower precision tokens gets this token type assigned.- See Also:
-
-
Constructor Details
-
NumericTokenStream
public NumericTokenStream()Creates a token stream for numeric values using the defaultprecisionStep
NumericUtils.PRECISION_STEP_DEFAULT
(4). The stream is not yet initialized, before using set a value using the various set???Value() methods. -
NumericTokenStream
public NumericTokenStream(int precisionStep) Creates a token stream for numeric values with the specifiedprecisionStep
. The stream is not yet initialized, before using set a value using the various set???Value() methods. -
NumericTokenStream
Expert: Creates a token stream for numeric values with the specifiedprecisionStep
using the givenAttributeSource.AttributeFactory
. The stream is not yet initialized, before using set a value using the various set???Value() methods.
-
-
Method Details
-
setLongValue
Initializes the token stream with the suppliedlong
value.- Parameters:
value
- the value, for which this TokenStream should enumerate tokens.- Returns:
- this instance, because of this you can use it the following way:
new Field(name, new NumericTokenStream(precisionStep).setLongValue(value))
-
setIntValue
Initializes the token stream with the suppliedint
value.- Parameters:
value
- the value, for which this TokenStream should enumerate tokens.- Returns:
- this instance, because of this you can use it the following way:
new Field(name, new NumericTokenStream(precisionStep).setIntValue(value))
-
setDoubleValue
Initializes the token stream with the supplieddouble
value.- Parameters:
value
- the value, for which this TokenStream should enumerate tokens.- Returns:
- this instance, because of this you can use it the following way:
new Field(name, new NumericTokenStream(precisionStep).setDoubleValue(value))
-
setFloatValue
Initializes the token stream with the suppliedfloat
value.- Parameters:
value
- the value, for which this TokenStream should enumerate tokens.- Returns:
- this instance, because of this you can use it the following way:
new Field(name, new NumericTokenStream(precisionStep).setFloatValue(value))
-
reset
public void reset()Description copied from class:TokenStream
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).- Overrides:
reset
in classTokenStream
-
incrementToken
public boolean incrementToken()Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
-
getPrecisionStep
public int getPrecisionStep()Returns the precision step.
-