Package org.apache.jackrabbit.oak.spi.blob

Class AbstractBlobStore

java.lang.Object
org.apache.jackrabbit.oak.spi.blob.AbstractBlobStore
All Implemented Interfaces:
AutoCloseable, Cache.Backend<AbstractBlobStore.BlockId,AbstractBlobStore.Data>, BlobStore, GarbageCollectableBlobStore
Direct Known Subclasses:
CachingBlobStore, FileBlobStore, MemoryBlobStore
public abstract class AbstractBlobStore extends Object implements GarbageCollectableBlobStore, Cache.Backend<AbstractBlobStore.BlockId,AbstractBlobStore.Data>
An abstract data store that splits the binaries in relatively small blocks, so that each block fits in memory.

Each data store id is a list of zero or more entries. Each entry is either

  • data (a number of bytes), or
  • the hash code of the content of a number of bytes, or
  • the hash code of the content of a data store id (indirect hash)
Thanks to the indirection, blocks can be kept relatively small, so that caching is simpler, and so that the storage backend doesn't need to support arbitrary size blobs (some storage backends buffer blobs in memory) and fast seeks (some storage backends re-read the whole blob when seeking).

The format of a 'data' entry is: type (one byte; 0 for data), length (variable size int), data (bytes).

The format of a 'hash of content' entry is: type (one byte; 1 for hash), level (variable size int, 0 meaning not nested), size (variable size long), hash code length (variable size int), hash code.

The format of a 'hash of data store id' entry is: type (one byte; 1 for hash), level (variable size int, nesting level), total size (variable size long), size of data store id (variable size long), hash code length (variable size int), hash code.

  • Field Details

    • HASH_ALGORITHM

      protected static final String HASH_ALGORITHM
      See Also:

    • TYPE_DATA

      protected static final int TYPE_DATA
      See Also:

    • TYPE_HASH

      protected static final int TYPE_HASH
      See Also:

    • BLOCK_SIZE_LIMIT

      protected static final int BLOCK_SIZE_LIMIT
      The minimum block size. Blocks must be larger than that so that the content hash is always shorter than the data itself.
      See Also:

    • inUse

      protected Map<String,WeakReference<String>> inUse
      The blob ids that are still floating around in memory. The blob store assumes such binaries must not be deleted, because those binaries are not referenced yet in a way the garbage collection algorithm can detect (not referenced at all, or only referenced in memory).

  • Constructor Details

    • AbstractBlobStore

      public AbstractBlobStore()

  • Method Details

    • setBlockSizeMin

      public void setBlockSizeMin(int x)

    • getBlockSizeMin

      public long getBlockSizeMin()
      Description copied from interface: GarbageCollectableBlobStore
      Get the minimum block size (if there is any).
      Specified by:
      getBlockSizeMin in interface GarbageCollectableBlobStore
      Returns:
      the block size

    • setBlockSize

      public void setBlockSize(int x)
      Description copied from interface: GarbageCollectableBlobStore
      Set the block size used by this blob store, if the blob store splits binaries into blocks. If not, this setting is ignored.
      Specified by:
      setBlockSize in interface GarbageCollectableBlobStore
      Parameters:
      x - the block size in bytes.

    • setStatsCollector

      public void setStatsCollector(BlobStatsCollector stats)

    • getStatsCollector

      protected BlobStatsCollector getStatsCollector()

    • getBlockSize

      public int getBlockSize()

    • writeBlob

      public String writeBlob(String tempFilePath) throws IOException
      Description copied from interface: GarbageCollectableBlobStore
      Write a blob from a temporary file. The temporary file is removed afterwards. A file based blob stores might simply rename the file, so that no additional writes are necessary.
      Specified by:
      writeBlob in interface GarbageCollectableBlobStore
      Parameters:
      tempFilePath - the temporary file name
      Returns:
      the blob id
      Throws:
      IOException

    • writeBlob

      public String writeBlob(InputStream in) throws IOException
      Description copied from interface: BlobStore
      Write a blob from an input stream. This method closes the input stream.
      Specified by:
      writeBlob in interface BlobStore
      Parameters:
      in - the input stream
      Returns:
      the blob id
      Throws:
      IOException

    • writeBlob

      public String writeBlob(InputStream in, BlobOptions options) throws IOException
      Default implementation ignores options and delegates to the writeBlob(InputStream) method.
      Specified by:
      writeBlob in interface BlobStore
      Parameters:
      in - the input stream to write
      options - the options to use
      Returns:
      Throws:
      IOException

    • getInputStream

      public InputStream getInputStream(String blobId) throws IOException
      Description copied from interface: BlobStore
      Returns a new stream for given blobId. The streams returned from multiple calls to this method are byte wise equals. That is, subsequent calls to read return the same sequence of bytes as long as neither call throws an exception.
      Specified by:
      getInputStream in interface BlobStore
      Parameters:
      blobId - the blob id
      Returns:
      a new stream for given blobId
      Throws:
      IOException

    • getReference

      public String getReference(@NotNull @NotNull String blobId)
      Description copied from interface: BlobStore
      Returns a secure reference to blob referred by blobid, or null if no such reference is available.
      Specified by:
      getReference in interface BlobStore
      Parameters:
      blobId - blobId referring the blob for which reference is required
      Returns:
      binary reference, or null

    • getBlobId

      public String getBlobId(@NotNull @NotNull String reference)
      Description copied from interface: BlobStore
      Returns the blobId that referred by the given binary reference. Returns null if the reference is invalid, for example if it points to a blob that does not exist.
      Specified by:
      getBlobId in interface BlobStore
      Parameters:
      reference - binary reference
      Returns:
      matching blobId, or null

    • getOrCreateReferenceKey

      protected byte[] getOrCreateReferenceKey()
      Returns the reference key of this blob store. If one does not already exist, it is automatically created in an implementation-specific way. The default implementation simply creates a temporary random key that's valid only until the data store gets restarted. Subclasses can override and/or decorate this method to support a more persistent reference key.

      This method is called only once during the lifetime of a data store instance and the return value is cached in memory, so it's no problem if the implementation is slow.

      Returns:
      reference key

    • setReferenceKey

      public void setReferenceKey(byte[] referenceKey)

    • setReferenceKeyEncoded

      public void setReferenceKeyEncoded(String encodedKey)
      Set the referenceKey from Base64 encoded byte array
      Parameters:
      encodedKey - base64 encoded key

    • setReferenceKeyPlainText

      public void setReferenceKeyPlainText(String textKey)
      Set the referenceKey from plain text. Key content would be UTF-8 encoding of the string.

      This is useful when setting key via generic bean property manipulation from string properties. User can specify the key in plain text and that would be passed on this object via PropertiesUtil.populate(Object, java.util.Map, boolean)

      Parameters:
      textKey - base64 encoded key
      See Also:

    • usesBlobId

      protected void usesBlobId(String blobId)

    • clearInUse

      public void clearInUse()
      Description copied from interface: GarbageCollectableBlobStore
      Clear all objects marked as "transiently in use".
      Specified by:
      clearInUse in interface GarbageCollectableBlobStore

    • storeBlock

      protected abstract void storeBlock(byte[] digest, int level, byte[] data) throws IOException
      Store a block of data.
      Parameters:
      digest - the content hash (32 bytes)
      level - the indirection level (0 is for user data, 1 is a list of digests that point to user data, 2 is a list of digests that point to digests, and so on). This parameter is for informational use only, and it is not required to store it unless that's easy to achieve
      data - the data to be stored (the number of bytes is at most the block size)
      Throws:
      IOException

    • startMark

      public abstract void startMark() throws IOException
      Description copied from interface: GarbageCollectableBlobStore
      Start the mark phase.
      Specified by:
      startMark in interface GarbageCollectableBlobStore
      Throws:
      IOException

    • sweep

      public abstract int sweep() throws IOException
      Description copied from interface: GarbageCollectableBlobStore
      Remove all unused blocks.
      Specified by:
      sweep in interface GarbageCollectableBlobStore
      Returns:
      the number of removed blocks
      Throws:
      IOException

    • isMarkEnabled

      protected abstract boolean isMarkEnabled()

    • mark

      protected abstract void mark(AbstractBlobStore.BlockId id) throws Exception
      Throws:
      Exception

    • markInUse

      protected void markInUse() throws IOException
      Throws:
      IOException

    • readBlob

      public int readBlob(String blobId, long pos, byte[] buff, int off, int length) throws IOException
      Description copied from interface: BlobStore
      Read a number of bytes from a blob.
      Specified by:
      readBlob in interface BlobStore
      Parameters:
      blobId - the blob id
      pos - the position within the blob
      buff - the target byte array
      off - the offset within the target array
      length - the number of bytes to read
      Returns:
      the number of bytes read
      Throws:
      IOException

    • load

      public AbstractBlobStore.Data load(AbstractBlobStore.BlockId id)
      Description copied from interface: Cache.Backend
      Load the object. The method does not need to be synchronized (it is synchronized in the cache)
      Specified by:
      load in interface Cache.Backend<AbstractBlobStore.BlockId,AbstractBlobStore.Data>
      Parameters:
      id - the key
      Returns:
      the value

    • readBlockFromBackend

      protected abstract byte[] readBlockFromBackend(AbstractBlobStore.BlockId id) throws Exception
      Load the block from the storage backend. Returns null if the block was not found.
      Parameters:
      id - the block id
      Returns:
      the block data, or null
      Throws:
      Exception

    • getBlobLength

      public long getBlobLength(String blobId) throws IOException
      Description copied from interface: BlobStore
      Get the length of the blob.
      Specified by:
      getBlobLength in interface BlobStore
      Parameters:
      blobId - the blob id
      Returns:
      the length
      Throws:
      IOException

    • mark

      protected void mark(String blobId) throws IOException
      Throws:
      IOException

    • resolveChunks

      public Iterator<String> resolveChunks(String blobId) throws IOException
      Description copied from interface: GarbageCollectableBlobStore
      Resolve chunks stored in the blob store from the given Id. This will not return any chunks stored in-line in the id.
      Specified by:
      resolveChunks in interface GarbageCollectableBlobStore
      Parameters:
      blobId - the blob id
      Returns:
      the iterator
      Throws:
      IOException - Signals that an I/O exception has occurred.

    • deleteChunks

      public boolean deleteChunks(List<String> chunkIds, long maxLastModifiedTime) throws Exception
      Description copied from interface: GarbageCollectableBlobStore
      Deletes the blobs with the given ids.
      Specified by:
      deleteChunks in interface GarbageCollectableBlobStore
      Parameters:
      chunkIds - the chunk ids
      maxLastModifiedTime - the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time
      Returns:
      true, if successful
      Throws:
      Exception - the exception

    • close

      public void close() throws Exception
      Specified by:
      close in interface AutoCloseable
      Throws:
      Exception