Class MMapDirectory
- All Implemented Interfaces:
Closeable
,AutoCloseable
Directory
implementation that uses
mmap for reading, and FSDirectory.FSIndexOutput
for writing.
NOTE: memory mapping uses up a portion of the
virtual memory address space in your process equal to the
size of the file being mapped. Before using this class,
be sure your have plenty of virtual address space, e.g. by
using a 64 bit JRE, or a 32 bit JRE with indexes that are
guaranteed to fit within the address space.
On 32 bit platforms also consult MMapDirectory(File, LockFactory, int)
if you have problems with mmap failing because of fragmented
address space. If you get an OutOfMemoryException, it is recommended
to reduce the chunk size, until it works.
Due to
this bug in Sun's JRE, MMapDirectory's IndexInput.close()
is unable to close the underlying OS file handle. Only when GC
finally collects the underlying objects, which could be quite
some time later, will the file handle be closed.
This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of.
This class supplies the workaround mentioned in the bug report
(see setUseUnmap(boolean)
), which may fail on
non-Sun JVMs. It forcefully unmaps the buffer on close by using
an undocumented internal cleanup functionality.
UNMAP_SUPPORTED
is true
, if the workaround
can be enabled (with no guarantees).
NOTE: Accessing this class either directly or
indirectly from a thread while it's interrupted can close the
underlying channel immediately if at the same time the thread is
blocked on IO. The channel will remain closed and subsequent access
to MMapDirectory
will throw a ClosedChannelException
.
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.store.FSDirectory
FSDirectory.FSIndexInput, FSDirectory.FSIndexOutput
Nested classes/interfaces inherited from class org.apache.lucene.store.Directory
Directory.IndexInputSlicer
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final int
Default max chunk size.static final boolean
true
, if this platform supports unmapping mmapped files.Fields inherited from class org.apache.lucene.store.FSDirectory
DEFAULT_READ_CHUNK_SIZE, directory, staleFiles
Fields inherited from class org.apache.lucene.store.BaseDirectory
isOpen, lockFactory
-
Constructor Summary
ConstructorsConstructorDescriptionMMapDirectory
(File path) Create a new MMapDirectory for the named location andNativeFSLockFactory
.MMapDirectory
(File path, LockFactory lockFactory) Create a new MMapDirectory for the named location.MMapDirectory
(File path, LockFactory lockFactory, int maxChunkSize) Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping. -
Method Summary
Modifier and TypeMethodDescriptioncreateSlicer
(String name, IOContext context) Creates anDirectory.IndexInputSlicer
for the given file name.final int
Returns the current mmap chunk size.boolean
Returnstrue
, if the unmap workaround is enabled.Creates an IndexInput for the file with the given name.void
setUseUnmap
(boolean useUnmapHack) This method enables the workaround for unmapping the buffers from address space after closingIndexInput
, that is mentioned in the bug report.Methods inherited from class org.apache.lucene.store.FSDirectory
close, createOutput, deleteFile, ensureCanWrite, fileExists, fileLength, fsync, getDirectory, getLockID, getReadChunkSize, listAll, listAll, onIndexOutputClosed, open, open, setLockFactory, setReadChunkSize, sync, toString
Methods inherited from class org.apache.lucene.store.BaseDirectory
clearLock, ensureOpen, getLockFactory, makeLock
-
Field Details
-
DEFAULT_MAX_BUFF
public static final int DEFAULT_MAX_BUFFDefault max chunk size.- See Also:
-
UNMAP_SUPPORTED
public static final boolean UNMAP_SUPPORTEDtrue
, if this platform supports unmapping mmapped files.
-
-
Constructor Details
-
MMapDirectory
Create a new MMapDirectory for the named location.- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default (NativeFSLockFactory
);- Throws:
IOException
- if there is a low-level I/O error
-
MMapDirectory
Create a new MMapDirectory for the named location andNativeFSLockFactory
.- Parameters:
path
- the path of the directory- Throws:
IOException
- if there is a low-level I/O error
-
MMapDirectory
Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping.- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default (NativeFSLockFactory
);maxChunkSize
- maximum chunk size (default is 1 GiBytes for 64 bit JVMs and 256 MiBytes for 32 bit JVMs) used for memory mapping.Especially on 32 bit platform, the address space can be very fragmented, so large index files cannot be mapped. Using a lower chunk size makes the directory implementation a little bit slower (as the correct chunk may be resolved on lots of seeks) but the chance is higher that mmap does not fail. On 64 bit Java platforms, this parameter should always be
1 << 30
, as the address space is big enough.Please note: The chunk size is always rounded down to a power of 2.
- Throws:
IOException
- if there is a low-level I/O error
-
-
Method Details
-
setUseUnmap
public void setUseUnmap(boolean useUnmapHack) This method enables the workaround for unmapping the buffers from address space after closingIndexInput
, that is mentioned in the bug report. This hack may fail on non-Sun JVMs. It forcefully unmaps the buffer on close by using an undocumented internal cleanup functionality.NOTE: Enabling this is completely unsupported by Java and may lead to JVM crashes if
IndexInput
is closed while another thread is still accessing it (SIGSEGV).- Throws:
IllegalArgumentException
- ifUNMAP_SUPPORTED
isfalse
and the workaround cannot be enabled.
-
getUseUnmap
public boolean getUseUnmap()Returnstrue
, if the unmap workaround is enabled.- See Also:
-
getMaxChunkSize
public final int getMaxChunkSize()Returns the current mmap chunk size.- See Also:
-
openInput
Creates an IndexInput for the file with the given name.- Specified by:
openInput
in classDirectory
- Throws:
IOException
-
createSlicer
Description copied from class:Directory
Creates anDirectory.IndexInputSlicer
for the given file name. IndexInputSlicer allows otherDirectory
implementations to efficiently open one or more slicedIndexInput
instances from a single file handle. The underlying file handle is kept open until theDirectory.IndexInputSlicer
is closed.- Must throw
FileNotFoundException
if the file does not exist (notjava.nio.file.NoSuchFileException
of Java 7).- Overrides:
createSlicer
in classDirectory
- Throws:
IOException
- if anIOException
occurs
- Must throw
-