org.apache.jackrabbit.oak.plugins.blob.datastore

Class DataStoreBlobStore

java.lang.Object

org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore

All Implemented Interfaces:

AutoCloseable, DataStore, BlobAccessProvider, BlobTrackingStore, TypedDataStore, SharedDataStore, BlobStore, GarbageCollectableBlobStore

Direct Known Subclasses:

SafeDataStoreBlobStore

public class DataStoreBlobStore
extends Object
implements DataStore, BlobStore, GarbageCollectableBlobStore, BlobTrackingStore, TypedDataStore, BlobAccessProvider

BlobStore wrapper for DataStore. Wraps Jackrabbit 2 DataStore and expose them as BlobStores It also handles inlining binaries if there size is smaller than DataStore.getMinRecordLength()

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataStoreBlobStore.BlobId

Nested classes/interfaces inherited from interface org.apache.jackrabbit.oak.plugins.blob.SharedDataStore

SharedDataStore.Type

Field Summary

Fields

Modifier and Type	Field and Description
protected LoadingCache<String,byte[]>	cache
static int	DEFAULT_CACHE_SIZE
protected DataStore	delegate
static String	MEM_CACHE_NAME
protected BlobStatsCollector	stats

Constructor Summary

Constructors

Constructor and Description
DataStoreBlobStore(DataStore delegate)
DataStoreBlobStore(DataStore delegate, boolean encodeLengthInId)
DataStoreBlobStore(DataStore delegate, boolean encodeLengthInId, int cacheSizeInMB)

Method Summary

Modifier and Type	Method and Description
void	addMetadataRecord(File f, String name) Adds the root record.
void	addMetadataRecord(InputStream stream, String name) Adds the root record.
DataRecord	addRecord(InputStream stream)
DataRecord	addRecord(InputStream input, BlobOptions options) Add a record with specified options.
void	addTracker(BlobTracker tracker) Registers a tracker in the data store.
void	clearCache() Clear the cache.
void	clearInUse() Clear all objects marked as "transiently in use".
void	close()
@Nullable Blob	completeBlobUpload(@NotNull String uploadToken) Complete a transaction for uploading a blob to a storage location via direct blob upload.
long	countDeleteChunks(List<String> chunkIds, long maxLastModifiedTime) Deletes the blobs with the given ids.
void	deleteAllMetadataRecords(String prefix) Deletes all records matching the given prefix.
int	deleteAllOlderThan(long min)
boolean	deleteChunks(List<String> chunkIds, long maxLastModifiedTime) Deletes the blobs with the given ids.
boolean	deleteMetadataRecord(String name) Deletes the root record represented by the given parameters.
protected String	extractBlobId(String encodedBlobId)
Iterator<String>	getAllChunkIds(long maxLastModifiedTime) Gets all the identifiers.
Iterator<DataIdentifier>	getAllIdentifiers()
List<DataRecord>	getAllMetadataRecords(String prefix) Gets the all root records.
Iterator<DataRecord>	getAllRecords() Retrieved an iterator over all DataRecords.
String	getBlobId(@NotNull String reference) Returns the blobId that referred by the given binary reference.
long	getBlobLength(String encodedBlobId) Get the length of the blob.
long	getBlockSizeMin() Get the minimum block size (if there is any).
CacheStats	getCacheStats()
protected DataRecord	getDataRecord(String blobId)
DataStore	getDataStore()
@Nullable URI	getDownloadURI(@NotNull Blob blob, @NotNull BlobDownloadOptions downloadOptions) Obtain a download URI for a Blob.
InputStream	getInputStream(String encodedBlobId) Returns a new stream for given blobId.
DataRecord	getMetadataRecord(String name) Retrieves the metadata record with the given name
int	getMinRecordLength()
DataRecord	getRecord(DataIdentifier identifier)
DataRecord	getRecordForId(DataIdentifier identifier) Retrieves the record for the given identifier
DataRecord	getRecordFromReference(String reference)
DataRecord	getRecordIfStored(DataIdentifier identifier)
String	getReference(@NotNull String encodedBlobId) Returns a secure reference to blob referred by blobid, or null if no such reference is available.
String	getRepositoryId() Returns the repository id (identifier for the repository in the DataStore)
protected InputStream	getStream(String blobId)
@Nullable BlobTracker	getTracker() Gets the traker registered in the data store.
SharedDataStore.Type	getType() Gets the type.
void	init(String homeDir)
@Nullable BlobUpload	initiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs) Begin a transaction to perform a direct blob upload to a storage location.
@Nullable BlobUpload	initiateBlobUpload(long maxUploadSizeInBytes, int maxNumberOfURIs, @NotNull BlobUploadOptions options) Begin a transaction to perform a direct blob upload to a storage location.
boolean	metadataRecordExists(String name) Checks if the metadata record with the name exists
int	readBlob(String encodedBlobId, long pos, byte[] buff, int off, int length) Read a number of bytes from a blob.
Iterator<String>	resolveChunks(String blobId) Resolve chunks stored in the blob store from the given Id.
void	setBlobStatsCollector(BlobStatsCollector stats)
void	setBlockSize(int x) Set the block size used by this blob store, if the blob store splits binaries into blocks.
void	setMaxCachedBinarySize(int maxCachedBinarySize)
void	setRepositoryId(String repositoryId) Sets the repository id to identify repository in the DataStore
void	startMark() Start the mark phase.
int	sweep() Remove all unused blocks.
String	toString()
void	updateModifiedDateOnAccess(long before)
String	writeBlob(InputStream stream) Write a blob from an input stream.
String	writeBlob(InputStream stream, BlobOptions options) Write a blob from an input stream with specified options.
String	writeBlob(String tempFileName) Write a blob from a temporary file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

delegate

protected final DataStore delegate

stats

protected BlobStatsCollector stats

cache

protected final LoadingCache<String,byte[]> cache

DEFAULT_CACHE_SIZE

public static final int DEFAULT_CACHE_SIZE

See Also:

Constant Field Values

MEM_CACHE_NAME

public static final String MEM_CACHE_NAME

See Also:

Constant Field Values

Constructor Detail

DataStoreBlobStore

public DataStoreBlobStore(DataStore delegate)

DataStoreBlobStore

public DataStoreBlobStore(DataStore delegate,
                          boolean encodeLengthInId)

DataStoreBlobStore

public DataStoreBlobStore(DataStore delegate,
                          boolean encodeLengthInId,
                          int cacheSizeInMB)

Method Detail

getRecordIfStored

public DataRecord getRecordIfStored(DataIdentifier identifier)
                             throws DataStoreException

Specified by:

getRecordIfStored in interface DataStore

Throws:

DataStoreException

getRecord

public DataRecord getRecord(DataIdentifier identifier)
                     throws DataStoreException

Specified by:

getRecord in interface DataStore

Throws:

DataStoreException

getRecordFromReference

public DataRecord getRecordFromReference(String reference)
                                  throws DataStoreException

Specified by:

getRecordFromReference in interface DataStore

Throws:

DataStoreException

addRecord

public DataRecord addRecord(InputStream stream)
                     throws DataStoreException

Specified by:

addRecord in interface DataStore

Throws:

DataStoreException

updateModifiedDateOnAccess

public void updateModifiedDateOnAccess(long before)

Specified by:

updateModifiedDateOnAccess in interface DataStore

deleteAllOlderThan

public int deleteAllOlderThan(long min)
                       throws DataStoreException

Specified by:

deleteAllOlderThan in interface DataStore

Throws:

DataStoreException

getAllIdentifiers

public Iterator<DataIdentifier> getAllIdentifiers()
                                           throws DataStoreException

Specified by:

getAllIdentifiers in interface DataStore

Throws:

DataStoreException

init

public void init(String homeDir)
          throws RepositoryException

Specified by:

init in interface DataStore

Throws:

RepositoryException

getMinRecordLength

public int getMinRecordLength()

Specified by:

getMinRecordLength in interface DataStore

close

public void close()
           throws DataStoreException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface DataStore

Throws:

DataStoreException

writeBlob

public String writeBlob(InputStream stream)
                 throws IOException

Description copied from interface: BlobStore

Write a blob from an input stream. This method closes the input stream.

Specified by:

writeBlob in interface BlobStore

Parameters:

stream - the input stream

Returns:

the blob id

Throws:

IOException

writeBlob

public String writeBlob(InputStream stream,
                        BlobOptions options)
                 throws IOException

Description copied from interface: BlobStore

Write a blob from an input stream with specified options. This method closes the input stream.

Specified by:

writeBlob in interface BlobStore

Parameters:

stream - the input stream to write

options - the options to use

Returns:

Throws:

IOException

readBlob

public int readBlob(String encodedBlobId,
                    long pos,
                    byte[] buff,
                    int off,
                    int length)
             throws IOException

Description copied from interface: BlobStore

Read a number of bytes from a blob.

Specified by:

readBlob in interface BlobStore

Parameters:

encodedBlobId - the blob id

pos - the position within the blob

buff - the target byte array

off - the offset within the target array

length - the number of bytes to read

Returns:

the number of bytes read

Throws:

IOException

getBlobLength

public long getBlobLength(String encodedBlobId)
                   throws IOException

Description copied from interface: BlobStore

Get the length of the blob.

Specified by:

getBlobLength in interface BlobStore

Parameters:

encodedBlobId - the blob id

Returns:

the length

Throws:

IOException

getBlobId

public String getBlobId(@NotNull
                        @NotNull String reference)

Description copied from interface: BlobStore

Returns the blobId that referred by the given binary reference. Returns null if the reference is invalid, for example if it points to a blob that does not exist.

Specified by:

getBlobId in interface BlobStore

Parameters:

reference - binary reference

Returns:

matching blobId, or null

getReference

public String getReference(@NotNull
                           @NotNull String encodedBlobId)

Description copied from interface: BlobStore

Returns a secure reference to blob referred by blobid, or null if no such reference is available.

Specified by:

getReference in interface BlobStore

Parameters:

encodedBlobId - blobId referring the blob for which reference is required

Returns:

binary reference, or null

getInputStream

public InputStream getInputStream(String encodedBlobId)
                           throws IOException

Description copied from interface: BlobStore

Returns a new stream for given blobId. The streams returned from multiple calls to this method are byte wise equals. That is, subsequent calls to read return the same sequence of bytes as long as neither call throws an exception.

Specified by:

getInputStream in interface BlobStore

Parameters:

encodedBlobId - the blob id

Returns:

a new stream for given blobId

Throws:

IOException

setBlockSize

public void setBlockSize(int x)

Description copied from interface: GarbageCollectableBlobStore

Set the block size used by this blob store, if the blob store splits binaries into blocks. If not, this setting is ignored.

Specified by:

setBlockSize in interface GarbageCollectableBlobStore

Parameters:

x - the block size in bytes.

writeBlob

public String writeBlob(String tempFileName)
                 throws IOException

Description copied from interface: GarbageCollectableBlobStore

Write a blob from a temporary file. The temporary file is removed afterwards. A file based blob stores might simply rename the file, so that no additional writes are necessary.

Specified by:

writeBlob in interface GarbageCollectableBlobStore

Parameters:

tempFileName - the temporary file name

Returns:

the blob id

Throws:

IOException

sweep

public int sweep()
          throws IOException

Description copied from interface: GarbageCollectableBlobStore

Remove all unused blocks.

Specified by:

sweep in interface GarbageCollectableBlobStore

Returns:

the number of removed blocks

Throws:

IOException

startMark

public void startMark()
               throws IOException

Description copied from interface: GarbageCollectableBlobStore

Start the mark phase.

Specified by:

startMark in interface GarbageCollectableBlobStore

Throws:

IOException

clearInUse

public void clearInUse()

Description copied from interface: GarbageCollectableBlobStore

Clear all objects marked as "transiently in use".

Specified by:

clearInUse in interface DataStore

Specified by:

clearInUse in interface GarbageCollectableBlobStore

clearCache

public void clearCache()

Description copied from interface: GarbageCollectableBlobStore

Clear the cache.

Specified by:

clearCache in interface GarbageCollectableBlobStore

getBlockSizeMin

public long getBlockSizeMin()

Description copied from interface: GarbageCollectableBlobStore

Get the minimum block size (if there is any).

Specified by:

getBlockSizeMin in interface GarbageCollectableBlobStore

Returns:

the block size

getAllChunkIds

public Iterator<String> getAllChunkIds(long maxLastModifiedTime)
                                throws Exception

Description copied from interface: GarbageCollectableBlobStore

Gets all the identifiers.

Specified by:

getAllChunkIds in interface GarbageCollectableBlobStore

Parameters:

maxLastModifiedTime - the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time

Returns:

the identifiers

Throws:

Exception - the exception

deleteChunks

public boolean deleteChunks(List<String> chunkIds,
                            long maxLastModifiedTime)
                     throws Exception

Description copied from interface: GarbageCollectableBlobStore

Deletes the blobs with the given ids.

Specified by:

deleteChunks in interface GarbageCollectableBlobStore

Parameters:

chunkIds - the chunk ids

maxLastModifiedTime - the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time

Returns:

true, if successful

Throws:

Exception - the exception

countDeleteChunks

public long countDeleteChunks(List<String> chunkIds,
                              long maxLastModifiedTime)
                       throws Exception

Description copied from interface: GarbageCollectableBlobStore

Deletes the blobs with the given ids.

Specified by:

countDeleteChunks in interface GarbageCollectableBlobStore

Parameters:

chunkIds - the chunk ids

maxLastModifiedTime - the max last modified time to consider for retrieval, with the special value '0' meaning no filtering by time

Returns:

long the count of successful deletions

Throws:

Exception - the exception

resolveChunks

public Iterator<String> resolveChunks(String blobId)
                               throws IOException

Description copied from interface: GarbageCollectableBlobStore

Resolve chunks stored in the blob store from the given Id. This will not return any chunks stored in-line in the id.

Specified by:

resolveChunks in interface GarbageCollectableBlobStore

Parameters:

blobId - the blob id

Returns:

the iterator

Throws:

IOException - Signals that an I/O exception has occurred.

addMetadataRecord

public void addMetadataRecord(InputStream stream,
                              String name)
                       throws DataStoreException

Description copied from interface: SharedDataStore

Adds the root record.

Specified by:

addMetadataRecord in interface SharedDataStore

Parameters:

stream - the stream

name - the name of the root record

Throws:

DataStoreException - the data store exception

addMetadataRecord

public void addMetadataRecord(File f,
                              String name)
                       throws DataStoreException

Description copied from interface: SharedDataStore

Adds the root record.

Specified by:

addMetadataRecord in interface SharedDataStore

Parameters:

f - the file

name - the name of the root record

Throws:

DataStoreException - the data store exception

getMetadataRecord

public DataRecord getMetadataRecord(String name)

Description copied from interface: SharedDataStore

Retrieves the metadata record with the given name

Specified by:

getMetadataRecord in interface SharedDataStore

Parameters:

name - the name of the record

Returns:

metadataRecordExists

public boolean metadataRecordExists(String name)

Description copied from interface: SharedDataStore

Checks if the metadata record with the name exists

Specified by:

metadataRecordExists in interface SharedDataStore

Returns:

getAllMetadataRecords

public List<DataRecord> getAllMetadataRecords(String prefix)

Description copied from interface: SharedDataStore

Gets the all root records.

Specified by:

getAllMetadataRecords in interface SharedDataStore

Returns:

the all root records

deleteMetadataRecord

public boolean deleteMetadataRecord(String name)

Description copied from interface: SharedDataStore

Deletes the root record represented by the given parameters.

Specified by:

deleteMetadataRecord in interface SharedDataStore

Parameters:

name - the name of the root record

Returns:

success/failure

deleteAllMetadataRecords

public void deleteAllMetadataRecords(String prefix)

Description copied from interface: SharedDataStore

Deletes all records matching the given prefix.

Specified by:

deleteAllMetadataRecords in interface SharedDataStore

Parameters:

prefix - metadata type identifier

setRepositoryId

public void setRepositoryId(String repositoryId)
                     throws DataStoreException

Description copied from interface: SharedDataStore

Sets the repository id to identify repository in the DataStore

Specified by:

setRepositoryId in interface SharedDataStore

Throws:

DataStoreException

getRepositoryId

public String getRepositoryId()

Description copied from interface: SharedDataStore

Returns the repository id (identifier for the repository in the DataStore)

Specified by:

getRepositoryId in interface SharedDataStore

Returns:

repository id

getAllRecords

public Iterator<DataRecord> getAllRecords()
                                   throws DataStoreException

Description copied from interface: SharedDataStore

Retrieved an iterator over all DataRecords.

Specified by:

getAllRecords in interface SharedDataStore

Returns:

iterator over DataRecords

Throws:

DataStoreException

getRecordForId

public DataRecord getRecordForId(DataIdentifier identifier)
                          throws DataStoreException

Description copied from interface: SharedDataStore

Retrieves the record for the given identifier

Specified by:

getRecordForId in interface SharedDataStore

Parameters:

identifier - the if of the record

Returns:

data record

Throws:

DataStoreException

getType

public SharedDataStore.Type getType()

Description copied from interface: SharedDataStore

Gets the type.

Specified by:

getType in interface SharedDataStore

Returns:

the type

addRecord

public DataRecord addRecord(InputStream input,
                            BlobOptions options)
                     throws DataStoreException

Description copied from interface: TypedDataStore

Add a record with specified options.

Specified by:

addRecord in interface TypedDataStore

Returns:

Throws:

DataStoreException

toString

public String toString()

Overrides:

toString in class Object

getDataStore

public DataStore getDataStore()

getCacheStats

public CacheStats getCacheStats()

setMaxCachedBinarySize

public void setMaxCachedBinarySize(int maxCachedBinarySize)

setBlobStatsCollector

public void setBlobStatsCollector(BlobStatsCollector stats)

addTracker

public void addTracker(BlobTracker tracker)

Description copied from interface: BlobTrackingStore

Registers a tracker in the data store.

Specified by:

addTracker in interface BlobTrackingStore

getTracker

@Nullable
public @Nullable BlobTracker getTracker()

Description copied from interface: BlobTrackingStore

Gets the traker registered in the data store.

Specified by:

getTracker in interface BlobTrackingStore

Returns:

tracker

getStream

protected InputStream getStream(String blobId)
                         throws IOException

Throws:

IOException

getDataRecord

protected DataRecord getDataRecord(String blobId)
                            throws DataStoreException

Throws:

DataStoreException

extractBlobId

protected String extractBlobId(String encodedBlobId)

initiateBlobUpload

@Nullable
public @Nullable BlobUpload initiateBlobUpload(long maxUploadSizeInBytes,
                                                         int maxNumberOfURIs)
                                                  throws IllegalArgumentException

Description copied from interface: BlobAccessProvider

Begin a transaction to perform a direct blob upload to a storage location. This method will throw IllegalArgumentException if no valid upload can be arranged with the arguments specified. E.g. the max upload size specified divided by the number of URIs requested indicates the minimum size of each upload. If that size exceeds the maximum upload size supported by the service provider, IllegalArgumentException is thrown.

Each service provider has specific limitations with regard to maximum upload sizes, maximum overall blob sizes, numbers of URIs in multi-part uploads, etc. which can lead to IllegalArgumentException being thrown. You should consult the documentation for your specific service provider for details.

Beyond service provider limitations, the implementation may also choose to enforce its own limitations and may throw this exception based on those limitations. Configuration may also be used to set limitations so this exception may be thrown when configuration parameters are exceeded.

Specified by:

initiateBlobUpload in interface BlobAccessProvider

Parameters:

maxUploadSizeInBytes - the largest size of the blob to be uploaded, in bytes, based on the caller's best guess. If the actual size of the file to be uploaded is known, that value should be used.

maxNumberOfURIs - the maximum number of URIs the client is able to accept. If the client does not support multi-part uploading, this value should be 1. Note that the implementing class is not required to support multi-part uploading so it may return only a single upload URI regardless of the value passed in for this parameter. If the client is able to accept any number of URIs, a value of -1 may be passed in to indicate that the implementation is free to return as many URIs as it desires.

Returns:

A BlobUpload referencing this direct upload, or null if the underlying implementation doesn't support direct uploading.

Throws:

IllegalArgumentException - if maxUploadSizeInBytes is not a positive value, or if maxNumberOfURIs is not either a positive value or -1, or if the upload cannot be completed as requested, due to a mismatch between the request parameters and the capabilities of the service provider or the implementation.

initiateBlobUpload

@Nullable
public @Nullable BlobUpload initiateBlobUpload(long maxUploadSizeInBytes,
                                                         int maxNumberOfURIs,
                                                         @NotNull
                                                         @NotNull BlobUploadOptions options)
                                                  throws IllegalArgumentException

Description copied from interface: BlobAccessProvider

Begin a transaction to perform a direct blob upload to a storage location. This method will throw IllegalArgumentException if no valid upload can be arranged with the arguments specified. E.g. the max upload size specified divided by the number of URIs requested indicates the minimum size of each upload. If that size exceeds the maximum upload size supported by the service provider, IllegalArgumentException is thrown.

Each service provider has specific limitations with regard to maximum upload sizes, maximum overall blob sizes, numbers of URIs in multi-part uploads, etc. which can lead to IllegalArgumentException being thrown. You should consult the documentation for your specific service provider for details.

Beyond service provider limitations, the implementation may also choose to enforce its own limitations and may throw this exception based on those limitations. Configuration may also be used to set limitations so this exception may be thrown when configuration parameters are exceeded.

Specified by:

initiateBlobUpload in interface BlobAccessProvider

Parameters:

maxUploadSizeInBytes - the largest size of the blob to be uploaded, in bytes, based on the caller's best guess. If the actual size of the file to be uploaded is known, that value should be used.

maxNumberOfURIs - the maximum number of URIs the client is able to accept. If the client does not support multi-part uploading, this value should be 1. Note that the implementing class is not required to support multi-part uploading so it may return only a single upload URI regardless of the value passed in for this parameter. If the client is able to accept any number of URIs, a value of -1 may be passed in to indicate that the implementation is free to return as many URIs as it desires.

options - an instance of BlobUploadOptions which allows the caller to specify any desired upload URI options.

Returns:

A BlobUpload referencing this direct upload, or null if the underlying implementation doesn't support direct uploading.

Throws:

IllegalArgumentException - if maxUploadSizeInBytes is not a positive value, or if maxNumberOfURIs is not either a positive value or -1, or if the upload cannot be completed as requested, due to a mismatch between the request parameters and the capabilities of the service provider or the implementation.

completeBlobUpload

@Nullable
public @Nullable Blob completeBlobUpload(@NotNull
                                                   @NotNull String uploadToken)
                                            throws IllegalArgumentException

Description copied from interface: BlobAccessProvider

Complete a transaction for uploading a blob to a storage location via direct blob upload.

This requires an uploadToken that can be obtained from the returned BlobUpload from a previous call to BlobAccessProvider.initiateBlobUpload(long, int). This token is required to complete the transaction for an upload to be valid and complete. The token includes encoded data about the transaction and may include a signature that will be verified by the implementation.

Specified by:

completeBlobUpload in interface BlobAccessProvider

Parameters:

uploadToken - the upload token from a BlobUpload object returned from a previous call to BlobAccessProvider.initiateBlobUpload(long, int).

Returns:

The Blob that was created, or null if the object could not be created.

Throws:

IllegalArgumentException - if the uploadToken is null, empty, or cannot be parsed or is otherwise invalid, e.g. if the included signature does not match.

getDownloadURI

@Nullable
public @Nullable URI getDownloadURI(@NotNull
                                              @NotNull Blob blob,
                                              @NotNull
                                              @NotNull BlobDownloadOptions downloadOptions)

Description copied from interface: BlobAccessProvider

Obtain a download URI for a Blob. This is usually a signed URI that can be used to directly download the blob corresponding to the provided Blob.

A caller must specify a BlobDownloadOptions instance. The implementation will attempt to apply the specified downloadOptions to the subsequent download. For example, if the caller knows that the URI refers to a specific type of content, the caller can specify that content type by setting it in the downloadOptions. The caller may also use a default instance obtained via BlobDownloadOptions.DEFAULT in which case the caller is indicating that the default behavior of the service provider is acceptable.

Specified by:

getDownloadURI in interface BlobAccessProvider

Parameters:

blob - The Blob to be downloaded.

downloadOptions - A BlobDownloadOptions instance that specifies any download options to be used for the download URI.

Returns:

A URI to download the blob directly or null if the blob cannot be downloaded directly.