DocumentNodeStore

The following DocumentNodeStore Counter metrics are available, independent of the underlying DocumentStore in use (MongoDB, RDB, etc.):

More information on CacheStats is also available in the CacheStatsMBean of the Oak API.

The following Meters are available, each exposing a cumulative count and various rates for the event (mean, one minute, five minute, fifteen minute):

Note: a merge operation on the DocumentNodeStore level is roughly equivalent to a JCR Session save() operation.

A DocumentNodeStore also exposes the following Timers with different percentiles for the duration of an operation. The timings include the latency between Oak and a backend store like MongoDB.

Note: Oak tracks the duration for above timers at millisecond or microsecond accuracy, but tools may translate the duration percentiles to different units.

If a DocumentNodeStore is configured with persistent caches, then the following Meters are available:

Note: the broadcast feature of the persistent cache is disabled by default, which means the number of received broadcast messages will be zero in that case.

Revision GC

Basic Revision GC can be monitored with the following Counters:

More detailed insight is provided by these Meters, each exposing a cumulative count and various rates for the event (mean, one minute, five minute, fifteen minute):

The following Timers are available for the Revision GC, with different percentiles for the duration of an operation. The timings include the latency between Oak and a backend store like MongoDB.

Note: Oak tracks the duration for above timers at microsecond accuracy, but many tools will translate the duration percentiles to milliseconds.

DocumentStore

Every DocumentStore exposes the following Meters with a cumulative count and various rates for each of the operations (mean, one minute, five minute, fifteen minute):

Note: the terms primary and slave are only relevant for a DocumentStore implementation that works with replicated data, like MongoDB. Implementations that do not make this distinction will report all calls for primary.

Note: update failures and retries are only relevant for the RDB DocumentStore. On a MongoDB DocumentStore, these values will always be zero.

In addition to above Meters, a DocumentStore also exposes the following Timers with different percentiles for the duration of an operation. The timings include the latency between Oak and a backend store like MongoDB.

Note: Oak tracks the duration for above timers at nanosecond accuracy, but many tools will translate the duration percentiles to milliseconds.

MongoDocumentStore

In addition to the general metrics of a DocumentStore, a MongoDocumentStore also exposes the following Counters:

Monitoring

While not all metrics described on this page are necessary to monitor a DocumentNodeStore, it is still advised to collect data for all metrics. In case of a problem, metrics can be a valuable source of historical data to analyze the problem and gain insight how the system behaved over time.

Some essential metrics that should be monitored are:

• Background read and write operations. These operations are scheduled to run every second. If this interval gets too long, changes from other cluster nodes will get picked up with a delay, which can result in failed merges because of conflicts. As a general rule of thumb, background operations that take longer than ten seconds should be analyzed. Possible reasons for an unusually long background operation may be increased load on the system with many changes to write to or read from the DocumentStore, general slow down of the JVM because of increased Java GC activity or an overloaded backend store.
• Background read lag rate. This is similar to the schedule of the background read and write operations. The one minute rate is usually below 1000 milliseconds and should be analyzed when this rate is higher than ten seconds.
• Lease update rate and duration. The DocumentNodeStore updates the lease roughly every 10 seconds. A lease update is a lightweight operation and should usually complete quickly. An increased update time may indicate a network problem or an overloaded backend store. A system should be analyzed when the lease update rate drops below 3 updates per minute and requires immediate attention when the rate is at one update per minute or lower. Please note, the DocumentNodeStore will shut itself down when the lease expires (the default lease time is two minutes).
• Revision GC rate and duration. Starting with Oak 1.8, Revision GC on MongoDB runs every five seconds. This is also known as Continuous Revision GC. This rate and the duration of a Revision GC cycle should be monitored and analyzed if the duration is more than five seconds for a couple of cycles.
• Revision GC failure. These should not occur and require immediate attention. The log files should provide details in this case why the Revision GC failed.
• Number of merges under an exclusive lock or that were suspended. Periods of increased merges with exclusive locks or suspended merges indicate conflicts usually caused by the application when multiple sessions try to write conflicting changes. The application should be analyzed to prevent such write patterns.