Class DataStoreBlobStore
- java.lang.Object
-
- org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore
-
- All Implemented Interfaces:
AutoCloseable,DataStore,BlobAccessProvider,BlobTrackingStore,TypedDataStore,SharedDataStore,BlobStore,GarbageCollectableBlobStore
public class DataStoreBlobStore extends Object implements DataStore, BlobStore, GarbageCollectableBlobStore, BlobTrackingStore, TypedDataStore, BlobAccessProvider
BlobStore wrapper for DataStore. Wraps Jackrabbit 2 DataStore and expose them as BlobStores It also handles inlining binaries if there size is smaller thanDataStore.getMinRecordLength()
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classDataStoreBlobStore.BlobId-
Nested classes/interfaces inherited from interface org.apache.jackrabbit.oak.plugins.blob.SharedDataStore
SharedDataStore.Type
-
-
Field Summary
Fields Modifier and Type Field Description protected org.apache.jackrabbit.guava.common.cache.LoadingCache<String,byte[]>cachestatic intDEFAULT_CACHE_SIZEprotected DataStoredelegatestatic StringMEM_CACHE_NAMEprotected BlobStatsCollectorstats
-
Constructor Summary
Constructors Constructor Description DataStoreBlobStore(DataStore delegate)DataStoreBlobStore(DataStore delegate, boolean encodeLengthInId)DataStoreBlobStore(DataStore delegate, boolean encodeLengthInId, int cacheSizeInMB)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidaddMetadataRecord(File f, String name)Adds the root record.voidaddMetadataRecord(InputStream stream, String name)Adds the root record.DataRecordaddRecord(InputStream stream)DataRecordaddRecord(InputStream input, BlobOptions options)Add a record with specified options.voidaddTracker(BlobTracker tracker)Registers a tracker in the data store.voidclearCache()Clear the cache.voidclearInUse()Clear all objects marked as "transiently in use".voidclose()@Nullable BlobcompleteBlobUpload(@NotNull String uploadToken)Complete a transaction for uploading a blob to a storage location via direct blob upload.longcountDeleteChunks(List<String> chunkIds, long maxLastModifiedTime)Deletes the blobs with the given ids.voiddeleteAllMetadataRecords(String prefix)Deletes all records matching the given prefix.intdeleteAllOlderThan(long min)booleandeleteChunks(List<String> chunkIds, long maxLastModifiedTime)Deletes the blobs with the given ids.booleandeleteMetadataRecord(String name)Deletes the root record represented by the given parameters.protected StringextractBlobId(String encodedBlobId)Iterator<String>getAllChunkIds(long maxLastModifiedTime)Gets all the identifiers.Iterator<DataIdentifier>getAllIdentifiers()List<DataRecord>getAllMetadataRecords(String prefix)Gets the all root records.Iterator<DataRecord>getAllRecords()Retrieved an iterator over all DataRecords.StringgetBlobId(@NotNull String reference)Returns the blobId that referred by the given binary reference.longgetBlobLength(String encodedBlobId)Get the length of the blob.longgetBlockSizeMin()Get the minimum block size (if there is any).CacheStatsgetCacheStats()protected DataRecordgetDataRecord(String blobId)DataStoregetDataStore()@Nullable URIgetDownloadURI(@NotNull Blob blob, @NotNull BlobDownloadOptions downloadOptions)Obtain a download URI for aBlob.InputStreamgetInputStream(String encodedBlobId)Returns a new stream for given blobId.DataRecordgetMetadataRecord(String name)Retrieves the metadata record with the given nameintgetMinRecordLength()DataRecordgetRecord(DataIdentifier identifier)DataRecordgetRecordForId(DataIdentifier identifier)Retrieves the record for the given identifierDataRecordgetRecordFromReference(String reference)DataRecordgetRecordIfStored(DataIdentifier identifier)StringgetReference(@NotNull String encodedBlobId)Returns a secure reference to blob referred by blobid, ornullif no such reference is available.StringgetRepositoryId()Returns the repository id (identifier for the repository in the DataStore)protected InputStreamgetStream(String blobId)@Nullable BlobTrackergetTracker()Gets the traker registered in the data store.SharedDataStore.TypegetType()Gets the type.voidinit(String homeDir)@Nullable BlobUploadinitiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs)Begin a transaction to perform a direct blob upload to a storage location.@Nullable BlobUploadinitiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs, @NotNull BlobUploadOptions options)Begin a transaction to perform a direct blob upload to a storage location.booleanmetadataRecordExists(String name)Checks if the metadata record with the name existsintreadBlob(String encodedBlobId, long pos, byte[] buff, int off, int length)Read a number of bytes from a blob.Iterator<String>resolveChunks(String blobId)Resolve chunks stored in the blob store from the given Id.voidsetBlobStatsCollector(BlobStatsCollector stats)voidsetBlockSize(int x)Set the block size used by this blob store, if the blob store splits binaries into blocks.voidsetMaxCachedBinarySize(int maxCachedBinarySize)voidsetRepositoryId(String repositoryId)Sets the repository id to identify repository in the DataStorevoidstartMark()Start the mark phase.intsweep()Remove all unused blocks.StringtoString()voidupdateModifiedDateOnAccess(long before)StringwriteBlob(InputStream stream)Write a blob from an input stream.StringwriteBlob(InputStream stream, BlobOptions options)Write a blob from an input stream with specified options.StringwriteBlob(String tempFileName)Write a blob from a temporary file.
-
-
-
Field Detail
-
delegate
protected final DataStore delegate
-
stats
protected BlobStatsCollector stats
-
cache
protected final org.apache.jackrabbit.guava.common.cache.LoadingCache<String,byte[]> cache
-
DEFAULT_CACHE_SIZE
public static final int DEFAULT_CACHE_SIZE
- See Also:
- Constant Field Values
-
MEM_CACHE_NAME
public static final String MEM_CACHE_NAME
- See Also:
- Constant Field Values
-
-
Method Detail
-
getRecordIfStored
public DataRecord getRecordIfStored(DataIdentifier identifier) throws DataStoreException
- Specified by:
getRecordIfStoredin interfaceDataStore- Throws:
DataStoreException
-
getRecord
public DataRecord getRecord(DataIdentifier identifier) throws DataStoreException
- Specified by:
getRecordin interfaceDataStore- Throws:
DataStoreException
-
getRecordFromReference
public DataRecord getRecordFromReference(String reference) throws DataStoreException
- Specified by:
getRecordFromReferencein interfaceDataStore- Throws:
DataStoreException
-
addRecord
public DataRecord addRecord(InputStream stream) throws DataStoreException
- Specified by:
addRecordin interfaceDataStore- Throws:
DataStoreException
-
updateModifiedDateOnAccess
public void updateModifiedDateOnAccess(long before)
- Specified by:
updateModifiedDateOnAccessin interfaceDataStore
-
deleteAllOlderThan
public int deleteAllOlderThan(long min) throws DataStoreException- Specified by:
deleteAllOlderThanin interfaceDataStore- Throws:
DataStoreException
-
getAllIdentifiers
public Iterator<DataIdentifier> getAllIdentifiers() throws DataStoreException
- Specified by:
getAllIdentifiersin interfaceDataStore- Throws:
DataStoreException
-
init
public void init(String homeDir) throws RepositoryException
- Specified by:
initin interfaceDataStore- Throws:
RepositoryException
-
getMinRecordLength
public int getMinRecordLength()
- Specified by:
getMinRecordLengthin interfaceDataStore
-
close
public void close() throws DataStoreException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceDataStore- Throws:
DataStoreException
-
writeBlob
public String writeBlob(InputStream stream) throws IOException
Description copied from interface:BlobStoreWrite a blob from an input stream. This method closes the input stream.- Specified by:
writeBlobin interfaceBlobStore- Parameters:
stream- the input stream- Returns:
- the blob id
- Throws:
IOException
-
writeBlob
public String writeBlob(InputStream stream, BlobOptions options) throws IOException
Description copied from interface:BlobStoreWrite a blob from an input stream with specified options. This method closes the input stream.- Specified by:
writeBlobin interfaceBlobStore- Parameters:
stream- the input stream to writeoptions- the options to use- Returns:
- Throws:
IOException
-
readBlob
public int readBlob(String encodedBlobId, long pos, byte[] buff, int off, int length) throws IOException
Description copied from interface:BlobStoreRead a number of bytes from a blob.- Specified by:
readBlobin interfaceBlobStore- Parameters:
encodedBlobId- the blob idpos- the position within the blobbuff- the target byte arrayoff- the offset within the target arraylength- the number of bytes to read- Returns:
- the number of bytes read
- Throws:
IOException
-
getBlobLength
public long getBlobLength(String encodedBlobId) throws IOException
Description copied from interface:BlobStoreGet the length of the blob.- Specified by:
getBlobLengthin interfaceBlobStore- Parameters:
encodedBlobId- the blob id- Returns:
- the length
- Throws:
IOException
-
getBlobId
public String getBlobId(@NotNull @NotNull String reference)
Description copied from interface:BlobStoreReturns the blobId that referred by the given binary reference. Returnsnullif the reference is invalid, for example if it points to a blob that does not exist.
-
getReference
public String getReference(@NotNull @NotNull String encodedBlobId)
Description copied from interface:BlobStoreReturns a secure reference to blob referred by blobid, ornullif no such reference is available.- Specified by:
getReferencein interfaceBlobStore- Parameters:
encodedBlobId- blobId referring the blob for which reference is required- Returns:
- binary reference, or
null
-
getInputStream
public InputStream getInputStream(String encodedBlobId) throws IOException
Description copied from interface:BlobStoreReturns a new stream for given blobId. The streams returned from multiple calls to this method are byte wise equals. That is, subsequent calls toreadreturn the same sequence of bytes as long as neither call throws an exception.- Specified by:
getInputStreamin interfaceBlobStore- Parameters:
encodedBlobId- the blob id- Returns:
- a new stream for given blobId
- Throws:
IOException
-
setBlockSize
public void setBlockSize(int x)
Description copied from interface:GarbageCollectableBlobStoreSet the block size used by this blob store, if the blob store splits binaries into blocks. If not, this setting is ignored.- Specified by:
setBlockSizein interfaceGarbageCollectableBlobStore- Parameters:
x- the block size in bytes.
-
writeBlob
public String writeBlob(String tempFileName) throws IOException
Description copied from interface:GarbageCollectableBlobStoreWrite a blob from a temporary file. The temporary file is removed afterwards. A file based blob stores might simply rename the file, so that no additional writes are necessary.- Specified by:
writeBlobin interfaceGarbageCollectableBlobStore- Parameters:
tempFileName- the temporary file name- Returns:
- the blob id
- Throws:
IOException
-
sweep
public int sweep() throws IOExceptionDescription copied from interface:GarbageCollectableBlobStoreRemove all unused blocks.- Specified by:
sweepin interfaceGarbageCollectableBlobStore- Returns:
- the number of removed blocks
- Throws:
IOException
-
startMark
public void startMark() throws IOExceptionDescription copied from interface:GarbageCollectableBlobStoreStart the mark phase.- Specified by:
startMarkin interfaceGarbageCollectableBlobStore- Throws:
IOException
-
clearInUse
public void clearInUse()
Description copied from interface:GarbageCollectableBlobStoreClear all objects marked as "transiently in use".- Specified by:
clearInUsein interfaceDataStore- Specified by:
clearInUsein interfaceGarbageCollectableBlobStore
-
clearCache
public void clearCache()
Description copied from interface:GarbageCollectableBlobStoreClear the cache.- Specified by:
clearCachein interfaceGarbageCollectableBlobStore
-
getBlockSizeMin
public long getBlockSizeMin()
Description copied from interface:GarbageCollectableBlobStoreGet the minimum block size (if there is any).- Specified by:
getBlockSizeMinin interfaceGarbageCollectableBlobStore- Returns:
- the block size
-
getAllChunkIds
public Iterator<String> getAllChunkIds(long maxLastModifiedTime) throws Exception
Description copied from interface:GarbageCollectableBlobStoreGets all the identifiers.- Specified by:
getAllChunkIdsin interfaceGarbageCollectableBlobStore- Parameters:
maxLastModifiedTime- the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time- Returns:
- the identifiers
- Throws:
Exception- the exception
-
deleteChunks
public boolean deleteChunks(List<String> chunkIds, long maxLastModifiedTime) throws Exception
Description copied from interface:GarbageCollectableBlobStoreDeletes the blobs with the given ids.- Specified by:
deleteChunksin interfaceGarbageCollectableBlobStore- Parameters:
chunkIds- the chunk idsmaxLastModifiedTime- the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time- Returns:
- true, if successful
- Throws:
Exception- the exception
-
countDeleteChunks
public long countDeleteChunks(List<String> chunkIds, long maxLastModifiedTime) throws Exception
Description copied from interface:GarbageCollectableBlobStoreDeletes the blobs with the given ids.- Specified by:
countDeleteChunksin interfaceGarbageCollectableBlobStore- Parameters:
chunkIds- the chunk idsmaxLastModifiedTime- the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time- Returns:
- long the count of successful deletions
- Throws:
Exception- the exception
-
resolveChunks
public Iterator<String> resolveChunks(String blobId) throws IOException
Description copied from interface:GarbageCollectableBlobStoreResolve chunks stored in the blob store from the given Id. This will not return any chunks stored in-line in the id.- Specified by:
resolveChunksin interfaceGarbageCollectableBlobStore- Parameters:
blobId- the blob id- Returns:
- the iterator
- Throws:
IOException- Signals that an I/O exception has occurred.
-
addMetadataRecord
public void addMetadataRecord(InputStream stream, String name) throws DataStoreException
Description copied from interface:SharedDataStoreAdds the root record.- Specified by:
addMetadataRecordin interfaceSharedDataStore- Parameters:
stream- the streamname- the name of the root record- Throws:
DataStoreException- the data store exception
-
addMetadataRecord
public void addMetadataRecord(File f, String name) throws DataStoreException
Description copied from interface:SharedDataStoreAdds the root record.- Specified by:
addMetadataRecordin interfaceSharedDataStore- Parameters:
f- the filename- the name of the root record- Throws:
DataStoreException- the data store exception
-
getMetadataRecord
public DataRecord getMetadataRecord(String name)
Description copied from interface:SharedDataStoreRetrieves the metadata record with the given name- Specified by:
getMetadataRecordin interfaceSharedDataStore- Parameters:
name- the name of the record- Returns:
-
metadataRecordExists
public boolean metadataRecordExists(String name)
Description copied from interface:SharedDataStoreChecks if the metadata record with the name exists- Specified by:
metadataRecordExistsin interfaceSharedDataStore- Returns:
-
getAllMetadataRecords
public List<DataRecord> getAllMetadataRecords(String prefix)
Description copied from interface:SharedDataStoreGets the all root records.- Specified by:
getAllMetadataRecordsin interfaceSharedDataStore- Returns:
- the all root records
-
deleteMetadataRecord
public boolean deleteMetadataRecord(String name)
Description copied from interface:SharedDataStoreDeletes the root record represented by the given parameters.- Specified by:
deleteMetadataRecordin interfaceSharedDataStore- Parameters:
name- the name of the root record- Returns:
- success/failure
-
deleteAllMetadataRecords
public void deleteAllMetadataRecords(String prefix)
Description copied from interface:SharedDataStoreDeletes all records matching the given prefix.- Specified by:
deleteAllMetadataRecordsin interfaceSharedDataStore- Parameters:
prefix- metadata type identifier
-
setRepositoryId
public void setRepositoryId(String repositoryId) throws DataStoreException
Description copied from interface:SharedDataStoreSets the repository id to identify repository in the DataStore- Specified by:
setRepositoryIdin interfaceSharedDataStore- Throws:
DataStoreException
-
getRepositoryId
public String getRepositoryId()
Description copied from interface:SharedDataStoreReturns the repository id (identifier for the repository in the DataStore)- Specified by:
getRepositoryIdin interfaceSharedDataStore- Returns:
- repository id
-
getAllRecords
public Iterator<DataRecord> getAllRecords() throws DataStoreException
Description copied from interface:SharedDataStoreRetrieved an iterator over all DataRecords.- Specified by:
getAllRecordsin interfaceSharedDataStore- Returns:
- iterator over DataRecords
- Throws:
DataStoreException
-
getRecordForId
public DataRecord getRecordForId(DataIdentifier identifier) throws DataStoreException
Description copied from interface:SharedDataStoreRetrieves the record for the given identifier- Specified by:
getRecordForIdin interfaceSharedDataStore- Parameters:
identifier- the if of the record- Returns:
- data record
- Throws:
DataStoreException
-
getType
public SharedDataStore.Type getType()
Description copied from interface:SharedDataStoreGets the type.- Specified by:
getTypein interfaceSharedDataStore- Returns:
- the type
-
addRecord
public DataRecord addRecord(InputStream input, BlobOptions options) throws DataStoreException
Description copied from interface:TypedDataStoreAdd a record with specified options.- Specified by:
addRecordin interfaceTypedDataStore- Returns:
- Throws:
DataStoreException
-
getDataStore
public DataStore getDataStore()
-
getCacheStats
public CacheStats getCacheStats()
-
setMaxCachedBinarySize
public void setMaxCachedBinarySize(int maxCachedBinarySize)
-
setBlobStatsCollector
public void setBlobStatsCollector(BlobStatsCollector stats)
-
addTracker
public void addTracker(BlobTracker tracker)
Description copied from interface:BlobTrackingStoreRegisters a tracker in the data store.- Specified by:
addTrackerin interfaceBlobTrackingStore
-
getTracker
@Nullable public @Nullable BlobTracker getTracker()
Description copied from interface:BlobTrackingStoreGets the traker registered in the data store.- Specified by:
getTrackerin interfaceBlobTrackingStore- Returns:
- tracker
-
getStream
protected InputStream getStream(String blobId) throws IOException
- Throws:
IOException
-
getDataRecord
protected DataRecord getDataRecord(String blobId) throws DataStoreException
- Throws:
DataStoreException
-
initiateBlobUpload
@Nullable public @Nullable BlobUpload initiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs) throws IllegalArgumentException
Description copied from interface:BlobAccessProviderBegin a transaction to perform a direct blob upload to a storage location. This method will throwIllegalArgumentExceptionif no valid upload can be arranged with the arguments specified. E.g. the max upload size specified divided by the number of URIs requested indicates the minimum size of each upload. If that size exceeds the maximum upload size supported by the service provider,IllegalArgumentExceptionis thrown.Each service provider has specific limitations with regard to maximum upload sizes, maximum overall blob sizes, numbers of URIs in multi-part uploads, etc. which can lead to
IllegalArgumentExceptionbeing thrown. You should consult the documentation for your specific service provider for details.Beyond service provider limitations, the implementation may also choose to enforce its own limitations and may throw this exception based on those limitations. Configuration may also be used to set limitations so this exception may be thrown when configuration parameters are exceeded.
- Specified by:
initiateBlobUploadin interfaceBlobAccessProvider- Parameters:
maxUploadSizeInBytes- the largest size of the blob to be uploaded, in bytes, based on the caller's best guess. If the actual size of the file to be uploaded is known, that value should be used.maxNumberOfURIs- the maximum number of URIs the client is able to accept. If the client does not support multi-part uploading, this value should be 1. Note that the implementing class is not required to support multi-part uploading so it may return only a single upload URI regardless of the value passed in for this parameter. If the client is able to accept any number of URIs, a value of -1 may be passed in to indicate that the implementation is free to return as many URIs as it desires.- Returns:
- A
BlobUploadreferencing this direct upload, ornullif the underlying implementation doesn't support direct uploading. - Throws:
IllegalArgumentException- ifmaxUploadSizeInBytesis not a positive value, or ifmaxNumberOfURIsis not either a positive value or -1, or if the upload cannot be completed as requested, due to a mismatch between the request parameters and the capabilities of the service provider or the implementation.
-
initiateBlobUpload
@Nullable public @Nullable BlobUpload initiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs, @NotNull @NotNull BlobUploadOptions options) throws IllegalArgumentException
Description copied from interface:BlobAccessProviderBegin a transaction to perform a direct blob upload to a storage location. This method will throwIllegalArgumentExceptionif no valid upload can be arranged with the arguments specified. E.g. the max upload size specified divided by the number of URIs requested indicates the minimum size of each upload. If that size exceeds the maximum upload size supported by the service provider,IllegalArgumentExceptionis thrown.Each service provider has specific limitations with regard to maximum upload sizes, maximum overall blob sizes, numbers of URIs in multi-part uploads, etc. which can lead to
IllegalArgumentExceptionbeing thrown. You should consult the documentation for your specific service provider for details.Beyond service provider limitations, the implementation may also choose to enforce its own limitations and may throw this exception based on those limitations. Configuration may also be used to set limitations so this exception may be thrown when configuration parameters are exceeded.
- Specified by:
initiateBlobUploadin interfaceBlobAccessProvider- Parameters:
maxUploadSizeInBytes- the largest size of the blob to be uploaded, in bytes, based on the caller's best guess. If the actual size of the file to be uploaded is known, that value should be used.maxNumberOfURIs- the maximum number of URIs the client is able to accept. If the client does not support multi-part uploading, this value should be 1. Note that the implementing class is not required to support multi-part uploading so it may return only a single upload URI regardless of the value passed in for this parameter. If the client is able to accept any number of URIs, a value of -1 may be passed in to indicate that the implementation is free to return as many URIs as it desires.options- an instance ofBlobUploadOptionswhich allows the caller to specify any desired upload URI options.- Returns:
- A
BlobUploadreferencing this direct upload, ornullif the underlying implementation doesn't support direct uploading. - Throws:
IllegalArgumentException- ifmaxUploadSizeInBytesis not a positive value, or ifmaxNumberOfURIsis not either a positive value or -1, or if the upload cannot be completed as requested, due to a mismatch between the request parameters and the capabilities of the service provider or the implementation.
-
completeBlobUpload
@Nullable public @Nullable Blob completeBlobUpload(@NotNull @NotNull String uploadToken) throws IllegalArgumentException
Description copied from interface:BlobAccessProviderComplete a transaction for uploading a blob to a storage location via direct blob upload.This requires an
uploadTokenthat can be obtained from the returnedBlobUploadfrom a previous call toBlobAccessProvider.initiateBlobUpload(long, int). This token is required to complete the transaction for an upload to be valid and complete. The token includes encoded data about the transaction and may include a signature that will be verified by the implementation.- Specified by:
completeBlobUploadin interfaceBlobAccessProvider- Parameters:
uploadToken- the upload token from aBlobUploadobject returned from a previous call toBlobAccessProvider.initiateBlobUpload(long, int).- Returns:
- The
Blobthat was created, ornullif the object could not be created. - Throws:
IllegalArgumentException- if theuploadTokenis null, empty, or cannot be parsed or is otherwise invalid, e.g. if the included signature does not match.
-
getDownloadURI
@Nullable public @Nullable URI getDownloadURI(@NotNull @NotNull Blob blob, @NotNull @NotNull BlobDownloadOptions downloadOptions)
Description copied from interface:BlobAccessProviderObtain a download URI for aBlob. This is usually a signed URI that can be used to directly download the blob corresponding to the providedBlob.A caller must specify a
BlobDownloadOptionsinstance. The implementation will attempt to apply the specifieddownloadOptionsto the subsequent download. For example, if the caller knows that the URI refers to a specific type of content, the caller can specify that content type by setting it in thedownloadOptions. The caller may also use a default instance obtained viaBlobDownloadOptions.DEFAULTin which case the caller is indicating that the default behavior of the service provider is acceptable.- Specified by:
getDownloadURIin interfaceBlobAccessProvider- Parameters:
blob- TheBlobto be downloaded.downloadOptions- ABlobDownloadOptionsinstance that specifies any download options to be used for the download URI.- Returns:
- A URI to download the blob directly or
nullif the blob cannot be downloaded directly.
-
-