Package org.apache.jackrabbit.core.data
Class CachingDataStore
- java.lang.Object
-
- org.apache.jackrabbit.core.data.AbstractDataStore
-
- org.apache.jackrabbit.core.data.CachingDataStore
-
- All Implemented Interfaces:
AsyncTouchCallback
,AsyncUploadCallback
,DataStore
,MultiDataStoreAware
- Direct Known Subclasses:
CachingFDS
,S3DataStore
,VFSDataStore
public abstract class CachingDataStore extends AbstractDataStore implements MultiDataStoreAware, AsyncUploadCallback, AsyncTouchCallback
A caching data store that consists ofLocalCache
andBackend
.Backend
is single source of truth. All methods first try to fetch information fromLocalCache
. If record is not available inLocalCache
, then it is fetched fromBackend
and saved toLocalCache
for further access. This class is designed to work withoutLocalCache
and then all information is fetched fromBackend
. To disableLocalCache
setsetCacheSize(long)
to 0. * Configuration:<DataStore class="org.apache.jackrabbit.aws.ext.ds.CachingDataStore"> <param name="
path
" value="/data/datastore"/> <param name="config
" value="${rep.home}/backend.properties"/> <param name="cacheSize
" value="68719476736"/> <param name="secret
" value="123456"/> <param name="setCachePurgeTrigFactor(double)
" value="0.95d"/> <param name="cacheSize
" value="0.85d"/> <param name="minRecordLength
" value="1024"/> <param name="continueOnAsyncUploadFailure
" value="false"/> <param name="concurrentUploadsThreads
" value="10"/> <param name="asyncUploadLimit
" value="100"/> <param name="uploadRetries
" value="3"/> <param name="touchAsync
" value="false"/> <param name="proactiveCaching
" value="true"/> <param name="recLengthCacheSize
" value="200"/> </DataStore>
-
-
Field Summary
Fields Modifier and Type Field Description protected Map<DataIdentifier,Long>
asyncDownloadCache
In memory map to hold in-progress asynchronous downloads.protected Map<DataIdentifier,Long>
asyncTouchCache
In memory map to hold in-progress asynchronous touch.protected Backend
backend
protected Map<DataIdentifier,WeakReference<DataIdentifier>>
inUse
All data identifiers that are currently in use are in this set until they are garbage collected.protected Map<DataIdentifier,Long>
recLenCache
In memory cache to holdDataRecord.getLength()
againstDataIdentifier
protected Map<DataIdentifier,Integer>
uploadRetryMap
In memory map to hold failed asynchronous uploadDataIdentifier
and its retry count.-
Fields inherited from class org.apache.jackrabbit.core.data.AbstractDataStore
DIGEST
-
-
Constructor Summary
Constructors Constructor Description CachingDataStore()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description DataRecord
addRecord(InputStream input)
Creates a new data record inBackend
.void
clearInUse()
Clear the in-use list.void
close()
Close the data storeboolean
confirmDelete(DataIdentifier identifier)
Method to confirm that identifier can be deleted fromBackend
protected abstract Backend
createBackend()
int
deleteAllOlderThan(long min)
Delete objects that have a modified date older than the specified date.void
deleteFromCache(DataIdentifier identifier)
void
deleteRecord(DataIdentifier identifier)
This method deletes record fromBackend
and then fromLocalCache
Iterator<DataIdentifier>
getAllIdentifiers()
Retrieves all identifiers fromBackend
.int
getAsyncUploadLimit()
Backend
getBackend()
double
getCachePurgeResizeFactor()
double
getCachePurgeTrigFactor()
long
getCacheSize()
int
getConcurrentUploadsThreads()
String
getConfig()
Return path of configuration properties.long
getLastModified(DataIdentifier identifier)
long
getLength(DataIdentifier identifier)
Return the length of record fromLocalCache
if available, otherwise retrieve it fromBackend
.protected abstract String
getMarkerFile()
int
getMinRecordLength()
Return mininum object length.protected byte[]
getOrCreateReferenceKey()
Returns the reference key of this data store.String
getPath()
Set<String>
getPendingUploads()
DataRecord
getRecord(DataIdentifier identifier)
Returns the identified data record.DataRecord
getRecordIfStored(DataIdentifier identifier)
Get a data record for the given identifier or null it data record doesn't exist inBackend
int
getUploadRetries()
void
init(String homeDir)
Initialized the data store.boolean
isContinueOnAsyncUploadFailure()
boolean
isInUse(DataIdentifier identifier)
void
onAbort(AsyncTouchResult result)
Callback method for aborted asynchronous touch.void
onAbort(AsyncUploadResult result)
Callback method for aborted asynchronous upload.void
onFailure(AsyncTouchResult result)
Callback method for failed asynchronous touch.void
onFailure(AsyncUploadResult result)
Callback method for failed asynchronous upload.void
onSuccess(AsyncTouchResult result)
Callback method for successful asynchronous touch.void
onSuccess(AsyncUploadResult result)
Callback method for successful asynchronous upload.void
setAsyncUploadLimit(int asyncUploadLimit)
void
setCachePurgeResizeFactor(double cachePurgeResizeFactor)
Set purge resize factor ofLocalCache
.void
setCachePurgeTrigFactor(double cachePurgeTrigFactor)
Set purge trigger factor ofLocalCache
.void
setCacheSize(long cacheSize)
Set size ofLocalCache
.void
setConcurrentUploadsThreads(int concurrentUploadsThreads)
void
setConfig(String config)
Set the configuration properties path.void
setContinueOnAsyncUploadFailure(boolean continueOnAsyncUploadFailure)
void
setMinRecordLength(int minRecordLength)
Set the minimum object length.void
setPath(String path)
Set path ofLocalCache
.void
setProactiveCaching(boolean proactiveCaching)
void
setRecLengthCacheSize(int recLengthCacheSize)
void
setSecret(String secret)
Setter for configuration based secretvoid
setTouchAsync(boolean touchAsync)
void
setUploadRetries(int uploadRetries)
void
updateModifiedDateOnAccess(long before)
From now on, update the modified date of an object even when accessing it.-
Methods inherited from class org.apache.jackrabbit.core.data.AbstractDataStore
encodeHexString, getRecordFromReference, getReferenceFromIdentifier
-
-
-
-
Field Detail
-
inUse
protected Map<DataIdentifier,WeakReference<DataIdentifier>> inUse
All data identifiers that are currently in use are in this set until they are garbage collected.
-
uploadRetryMap
protected final Map<DataIdentifier,Integer> uploadRetryMap
In memory map to hold failed asynchronous uploadDataIdentifier
and its retry count. Once if all retries are exhausted or file is successfully uploaded, then corresponding entry is flushed from the map. As all failed uploads are synchronously uploaded at startup, this map is not required to be persisted.
-
asyncTouchCache
protected final Map<DataIdentifier,Long> asyncTouchCache
In memory map to hold in-progress asynchronous touch. Once touch is successful corresponding entry is flushed from the map.
-
asyncDownloadCache
protected final Map<DataIdentifier,Long> asyncDownloadCache
In memory map to hold in-progress asynchronous downloads. Once download is finished corresponding entry is flushed from the map.
-
recLenCache
protected Map<DataIdentifier,Long> recLenCache
In memory cache to holdDataRecord.getLength()
againstDataIdentifier
-
backend
protected Backend backend
-
-
Method Detail
-
createBackend
protected abstract Backend createBackend()
-
getMarkerFile
protected abstract String getMarkerFile()
-
init
public void init(String homeDir) throws RepositoryException
Initialized the data store. If the path is not set, <repository home>/repository/datastore is used. This directory is automatically created if it does not yet exist. During first initialization, it upload all files from local datastore to backed and local datastore act as a local cache.- Specified by:
init
in interfaceDataStore
- Parameters:
homeDir
- the home directory of the repository- Throws:
RepositoryException
-
addRecord
public DataRecord addRecord(InputStream input) throws DataStoreException
Creates a new data record inBackend
. The stream is first consumed and the contents are saved in a temporary file and theAbstractDataStore.DIGEST
message digest of the stream is calculated. If a record with the sameAbstractDataStore.DIGEST
digest (and length) is found then it is returned. Otherwise new record is created inBackend
and the temporary file is moved in place toLocalCache
.- Specified by:
addRecord
in interfaceDataStore
- Parameters:
input
- binary stream- Returns:
CachingDataRecord
- Throws:
DataStoreException
- if the record could not be created.
-
getRecord
public DataRecord getRecord(DataIdentifier identifier) throws DataStoreException
Description copied from interface:DataStore
Returns the identified data record. The given identifier should be the identifier of a previously saved data record. Since records are never removed, there should never be cases where the identified record is not found. Abnormal cases like that are treated as errors and handled by throwing an exception.- Specified by:
getRecord
in interfaceDataStore
- Overrides:
getRecord
in classAbstractDataStore
- Parameters:
identifier
- data identifier- Returns:
- identified data record
- Throws:
DataStoreException
- if the data store could not be accessed, or if the given identifier is invalid
-
getRecordIfStored
public DataRecord getRecordIfStored(DataIdentifier identifier) throws DataStoreException
Get a data record for the given identifier or null it data record doesn't exist inBackend
- Specified by:
getRecordIfStored
in interfaceDataStore
- Parameters:
identifier
- identifier of record.- Returns:
- the
CachingDataRecord
or null. - Throws:
DataStoreException
- if the data store could not be accessed
-
updateModifiedDateOnAccess
public void updateModifiedDateOnAccess(long before)
Description copied from interface:DataStore
From now on, update the modified date of an object even when accessing it. Usually, the modified date is only updated when creating a new object, or when a new link is added to an existing object. When this setting is enabled, even getLength() will update the modified date.- Specified by:
updateModifiedDateOnAccess
in interfaceDataStore
- Parameters:
before
- - update the modified date to the current time if it is older than this value
-
getAllIdentifiers
public Iterator<DataIdentifier> getAllIdentifiers() throws DataStoreException
Retrieves all identifiers fromBackend
.- Specified by:
getAllIdentifiers
in interfaceDataStore
- Returns:
- an iterator over all DataIdentifier objects
- Throws:
DataStoreException
- if the list could not be read
-
deleteRecord
public void deleteRecord(DataIdentifier identifier) throws DataStoreException
This method deletes record fromBackend
and then fromLocalCache
- Specified by:
deleteRecord
in interfaceMultiDataStoreAware
- Parameters:
identifier
- data identifier- Throws:
DataStoreException
- if the data store could not be accessed, or if the given identifier is invalid
-
deleteAllOlderThan
public int deleteAllOlderThan(long min) throws DataStoreException
Description copied from interface:DataStore
Delete objects that have a modified date older than the specified date.- Specified by:
deleteAllOlderThan
in interfaceDataStore
- Parameters:
min
- the minimum time- Returns:
- the number of data records deleted
- Throws:
DataStoreException
-
getLastModified
public long getLastModified(DataIdentifier identifier) throws DataStoreException
- Throws:
DataStoreException
-
getLength
public long getLength(DataIdentifier identifier) throws DataStoreException
Return the length of record fromLocalCache
if available, otherwise retrieve it fromBackend
.- Throws:
DataStoreException
-
getOrCreateReferenceKey
protected byte[] getOrCreateReferenceKey() throws DataStoreException
Description copied from class:AbstractDataStore
Returns the reference key of this data store. If one does not already exist, it is automatically created in an implementation-specific way. The default implementation simply creates a temporary random key that's valid only until the data store gets restarted. Subclasses can override and/or decorate this method to support a more persistent reference key.This method is called only once during the lifetime of a data store instance and the return value is cached in memory, so it's no problem if the implementation is slow.
- Overrides:
getOrCreateReferenceKey
in classAbstractDataStore
- Returns:
- reference key
- Throws:
DataStoreException
- if the key is not available
-
deleteFromCache
public void deleteFromCache(DataIdentifier identifier) throws DataStoreException
- Throws:
DataStoreException
-
onSuccess
public void onSuccess(AsyncUploadResult result)
Description copied from interface:AsyncUploadCallback
Callback method for successful asynchronous upload.- Specified by:
onSuccess
in interfaceAsyncUploadCallback
-
onFailure
public void onFailure(AsyncUploadResult result)
Description copied from interface:AsyncUploadCallback
Callback method for failed asynchronous upload.- Specified by:
onFailure
in interfaceAsyncUploadCallback
-
onAbort
public void onAbort(AsyncUploadResult result)
Description copied from interface:AsyncUploadCallback
Callback method for aborted asynchronous upload.- Specified by:
onAbort
in interfaceAsyncUploadCallback
-
onSuccess
public void onSuccess(AsyncTouchResult result)
Description copied from interface:AsyncTouchCallback
Callback method for successful asynchronous touch.- Specified by:
onSuccess
in interfaceAsyncTouchCallback
-
onFailure
public void onFailure(AsyncTouchResult result)
Description copied from interface:AsyncTouchCallback
Callback method for failed asynchronous touch.- Specified by:
onFailure
in interfaceAsyncTouchCallback
-
onAbort
public void onAbort(AsyncTouchResult result)
Description copied from interface:AsyncTouchCallback
Callback method for aborted asynchronous touch.- Specified by:
onAbort
in interfaceAsyncTouchCallback
-
confirmDelete
public boolean confirmDelete(DataIdentifier identifier)
Method to confirm that identifier can be deleted fromBackend
- Parameters:
identifier
-- Returns:
-
clearInUse
public void clearInUse()
Description copied from interface:DataStore
Clear the in-use list. This is only used for testing to make the the garbage collection think that objects are no longer in use.- Specified by:
clearInUse
in interfaceDataStore
-
isInUse
public boolean isInUse(DataIdentifier identifier)
-
close
public void close() throws DataStoreException
Description copied from interface:DataStore
Close the data store- Specified by:
close
in interfaceDataStore
- Throws:
DataStoreException
- if a problem occurred
-
setSecret
public void setSecret(String secret)
Setter for configuration based secret- Parameters:
secret
- the secret used to sign reference binaries
-
setMinRecordLength
public void setMinRecordLength(int minRecordLength)
Set the minimum object length.- Parameters:
minRecordLength
- the length
-
getMinRecordLength
public int getMinRecordLength()
Return mininum object length.- Specified by:
getMinRecordLength
in interfaceDataStore
- Returns:
- the minimum size in bytes
-
getConfig
public String getConfig()
Return path of configuration properties.- Returns:
- path of configuration properties.
-
setConfig
public void setConfig(String config)
Set the configuration properties path.- Parameters:
config
- path of configuration properties.
-
getCacheSize
public long getCacheSize()
- Returns:
- size of
LocalCache
.
-
setCacheSize
public void setCacheSize(long cacheSize)
Set size ofLocalCache
.- Parameters:
cacheSize
- size ofLocalCache
.
-
getPath
public String getPath()
- Returns:
- path of
LocalCache
.
-
setPath
public void setPath(String path)
Set path ofLocalCache
.- Parameters:
path
- ofLocalCache
.
-
getCachePurgeTrigFactor
public double getCachePurgeTrigFactor()
- Returns:
- Purge trigger factor of
LocalCache
.
-
setCachePurgeTrigFactor
public void setCachePurgeTrigFactor(double cachePurgeTrigFactor)
Set purge trigger factor ofLocalCache
.- Parameters:
cachePurgeTrigFactor
- purge trigger factor.
-
getCachePurgeResizeFactor
public double getCachePurgeResizeFactor()
- Returns:
- Purge resize factor of
LocalCache
.
-
setCachePurgeResizeFactor
public void setCachePurgeResizeFactor(double cachePurgeResizeFactor)
Set purge resize factor ofLocalCache
.- Parameters:
cachePurgeResizeFactor
- purge resize factor.
-
getConcurrentUploadsThreads
public int getConcurrentUploadsThreads()
-
setConcurrentUploadsThreads
public void setConcurrentUploadsThreads(int concurrentUploadsThreads)
-
getAsyncUploadLimit
public int getAsyncUploadLimit()
-
setAsyncUploadLimit
public void setAsyncUploadLimit(int asyncUploadLimit)
-
isContinueOnAsyncUploadFailure
public boolean isContinueOnAsyncUploadFailure()
-
setContinueOnAsyncUploadFailure
public void setContinueOnAsyncUploadFailure(boolean continueOnAsyncUploadFailure)
-
getUploadRetries
public int getUploadRetries()
-
setUploadRetries
public void setUploadRetries(int uploadRetries)
-
setTouchAsync
public void setTouchAsync(boolean touchAsync)
-
setProactiveCaching
public void setProactiveCaching(boolean proactiveCaching)
-
setRecLengthCacheSize
public void setRecLengthCacheSize(int recLengthCacheSize)
-
getBackend
public Backend getBackend()
-
-