Oak Documentation

SegmentNodeStore (TarMK)

By default SegmentNodeStore (aka TarMK) does not require a BlobStore. Instead the binary content is directly stored as part of segment blob itself. Depending on requirements one of the following can be used

• FileDataStore - This should be used if the blobs/binaries have to be shared between multiple repositories. This would also be used when a JR2 repository is migrated to Oak
• S3DataStore - This should be used when binaries are stored in Amazon S3
• AzureDataStore - This should be used when binaries are stored in Microsoft Azure Blob storage

DocumentNodeStore

By default DocumentNodeStore when running on Mongo uses MongoBlobStore. Depending on requirements one of the following can be used

• MongoBlobStore - Used by default and recommended only for development and testing.
• FileDataStore - This should be used if the binaries have to be stored on the file system. This would also be used when a JR2 repository is migrated to Oak
• S3DataStore - This should be used when binaries are stored in Amazon S3. Typically used when running in Amazon AWS
• AzureDataStore - This should be used when binaries are stored in Microsoft Azure Blob storage

Caching DataStore

The DataStore implementations S3DataStore,CachingFileDataStore and AzureDataStore support local file system caching for the files/blobs and extend the AbstractSharedCachingDataStore class which implements the caching functionality. The CachingFileDataStore is useful when the DataStore is on nfs. The cache has a size limit and is configured by the cacheSize parameter.

Downloads

The local cache will be checked for existence of the record corresponding to the requested file/blob before accessing it from the DataStore. When the cache exceeds the limit configured while adding a file into the cache then some of the file(s) will be evicted to reclaim space.

Asynchronous Uploads

The cache also supports asynchronous uploads to the DataStore. The files are staged locally in the cache on the file system and an asynchronous job started to upload the file. The number of asynchronous uploads are limited by the size of staging cache configured by the stagingSplitPercentage parameter and is by default set to 10. This defines the ratio of the cacheSize to be dedicated for the staging cache. The percentage of cache available for downloads is calculated as (100 - stagingSplitPerentage) * cacheSize (by default 90). The asynchronous uploads are also multi-threaded and is governed by the uploadThreads configuration parameter. The default value is 10.

The files are moved to the main download cache after the uploads are complete. When the staging cache exceeds the limit, the files are uploaded synchronously to the DataStore until the previous asynchronous uploads are complete and space available in the staging cache. The uploaded files are removed from the staging area by a periodic job whose interval is configured by the stagingPurgeInterval configuration parameter. The default value is 300 seconds.

Any failed uploads (due to various reasons e.g. network disruption) are put on a retry queue and retried periodically with the configured interval stagingRetryInterval. The default value for is 600 seconds.

Caching Stats

The ConsolidatedDataStoreCacheStats is registered as an MBean and provides a snapshot of the cache performance for both the download and the upload staging cache.

The following table explains the different statistics exposed for both type of caches

The parameters above can be used to size the cache. For example:

• The hitRate is an important parameter and if much below 1 then indicates that the cache is low based on the load and should be increased.
• If the staging cache has a low hit ratio, download cache has a high hit ratio and also its current size is much less than the maxSize then its better to increase the stagingSplitPercentage parameter.

The MBean also exposes a method isFileSynced which takes a node path of a binary and returns whether the associated file/blob has been uploaded to the DataStore.

Upgrade (Pre Oak 1.6 caching)

When upgrading from the older cache implementation the process should be seamless and any pending uploads would be scheduled for upload and any previously downloaded files in the cache will be put in the cache on initialization. There is a slight difference in the structure of the local file system cache directory. Whereas in the older cache structure both the downloaded and the upload files were put directly under the cache path. The newer structure segregates the downloads and uploads and stores them under cache path under the directories download and upload
respectively.

There is also an option to upgrade the cache offline by using the datastorecacheupgrade command of oak-run. The details on how to execute the command and the different parameters can be checked in the readme for the oak-run module.

Caching of Blob ids locally (Oak 1.6.x)

For the FileDataStore, S3DataStore and AzureDataStore the blob ids are cached locally on the disk when they are created which speeds up the ‘Mark BlobStore’ phase. The locally tracked ids are synchronized with the data store periodically to enable other cluster nodes or different repositories sharing the datastore to get a consolidated list of all blob ids. The interval of synchronization is defined by the OSGi configuration parameter blobTrackSnapshotIntervalInSecs for the configured NodeStore services.

If 2 garbage collection cycles are executed within the blobTrackSnapshotIntervalInSecs then there may be warnings in the logs of some missing blob ids which is due to the fact that the deletions due to earlier gc has not been synchronized with the data store. It's ok to either ignore these warnings or to adjust the blobTrackSnapshotIntervalInSecs parameter according to the schedule identified for running blob gc.

When upgrading an existing system to take advantage of caching the existing blob ids have to be cached. One of the following should be executed.

• Use MarkSweepGarbageCollectot#collectGarbage(boolean markOnly, boolean forceBlobRetrieve) with true for forceBlobRetrieve parameter to force retrieving blob ids from the datastore and cache locally also.
• Execute Blob GC before the configured time duration of blobTrackSnapshotIntervalInSecs.
• Execute consistency check from the JMX BlobGCMbean before the configured time duration of blobTrackSnapshotIntervalInSecs.
• Execute datastorecheck command offline using oak-run with the --track option as defined in consistency check.

Shared DataStore Blob Garbage Collection (Since 1.2.0)

import java.util.Hashtable;

import javax.management.openmbean.TabularData;
import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.ObjectName;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.management.openmbean.CompositeData;


/**
 * Checks the status of the mark operation on all instances sharing the DataStore.
 */
public class GetGCStats {

    public static void main(String[] args) throws Exception {
        String userid = "<user>";
        String password = "<password>";
        String serverUrl = "service:jmx:rmi:///jndi/rmi://<host:port>/jmxrmi";
        String OBJECT_NAME = "org.apache.jackrabbit.oak:name=Document node store blob garbage collection,type=BlobGarbageCollection";
        String[] buffer = new String[] {userid, password};
        Hashtable<String, String[]> attributes = new Hashtable<String, String[]>();
        attributes.put("jmx.remote.credentials", buffer);
        MBeanServerConnection server = JMXConnectorFactory
            .connect(new JMXServiceURL(serverUrl), attributes).getMBeanServerConnection();
        ObjectName name = new ObjectName(OBJECT_NAME);
        BlobGCMBean gcBean = MBeanServerInvocationHandler
            .newProxyInstance(server, name, BlobGCMBean.class, false);

        boolean markDone = checkMarkDone("GlobalMarkStats", gcBean.getGlobalMarkStats());
        System.out.println("Mark done on all instances - " + markDone);
    }

    public static boolean checkMarkDone(String operation, TabularData data) {
        System.out.println("-----Operation " + operation + "--------------");

        boolean markDoneOnOthers = true;
        try {
            System.out.println("Number of instances " + data.size());

            for (Object o : data.values()) {
                CompositeData row = (CompositeData) o;
                String repositoryId = row.get("repositoryId").toString();
                System.out.println("Repository  " + repositoryId);

                if ((!row.containsKey("markEndTime")
                        || row.get("markEndTime") == null
                        || row.get("markEndTime").toString().length() == 0)) {
                    markDoneOnOthers = false;
                    System.out.println("Mark not done on repository : " + repositoryId);
                }
            }
        } catch (Exception e) {
            System.out.println(
                "-----Error during operation " + operation + "--------------" + e.getMessage());
        }
        System.out.println("-----Completed " + operation + "--------------");

        return markDoneOnOthers;
    }
}

Consistency Check

The data store consistency check will report any data store binaries that are missing but are still referenced. The consistency check can be triggered by:

• MarkSweepGarbageCollector#checkConsistency
• If the MBeans are registered in the MBeanServer then the following can also be used:
  • BlobGCMbean#checkConsistency

After the consistency check is complete, a message will show the number of binaries reported as missing. If the number is greater than 0, check the logs configured for org.apache.jackrabbit.oak.plugins.blob .MarkSweepGarbageCollector for more details on the missing binaries.

Below is an example of how the missing binaries are reported in the logs:

11:32:39.673 INFO [main] MarkSweepGarbageCollector.java:600 Consistency check found 1 missing blobs 11:32:39.673 WARN [main] MarkSweepGarbageCollector.java:602 Consistency check failure in the the blob store : DataStore backed BlobStore [org.apache.jackrabbit.oak.plugins.blob.datastore.OakFileDataStore], check missing candidates in file /tmp/gcworkdir-1467352959243/gccand-1467352959243

• datastorecheck command of oak-run can also be used to execute a consistency check on the datastore. The details on how to execute the command and the different parameters can be checked in the readme for the oak-run module.