Performance tuning
Performance requirements
Your key performance requirement is to satisfy your users or customers with the resources available to you. Before you can solve potential performance problems, define what those users or customers expect. Determine which resources you will have to satisfy their expectations.
Service level objectives
A service level objective (SLO) is a target for a directory service level that you can measure quantitatively. If possible, base SLOs on what your key users expect from the service in terms of performance.
Define SLOs for at least the following areas:
-
Directory service response times
Directory service response times range from less than a millisecond on average, across a low latency connection on the same network, to however long it takes your network to deliver the response.
More important than average or best response times is the response time distribution, because applications set timeouts based on worst case scenarios.
An example response time performance requirement is, Directory response times must average less than 10 milliseconds for all operations except searches returning more than 10 entries, with 99.9% of response times under 40 milliseconds.
-
Directory service throughput
Directories can serve many thousands of operations per second. In fact there is no upper limit for read operations such as searches, because only write operations must be replicated. To increase read throughput, simply add additional replicas.
More important than average throughput is peak throughput. You might have peak write throughput in the middle of the night when batch jobs update entries in bulk, and peak binds for a special event or first thing Monday morning.
An example throughput performance requirement is, The directory service must sustain a mix of 5,000 operations per second made up of 70% reads, 25% modifies, 3% adds, and 2% deletes.
Ideally, you mimic the behavior of key operations during performance testing, so that you understand the patterns of operations in the throughput you need to provide.
-
Directory service availability
DS software is designed to let you build directory services that are basically available, including during maintenance and even upgrade of individual servers.
To reach very high levels of availability, you must also ensure that your operations execute in a way that preserves availability.
Availability requirements can be as lax as a best effort, or as stringent as 99.999% or more uptime.
Replication is the DS feature that allows you to build a highly available directory service.
-
Directory service administrative support
Be sure to understand how you support your users when they run into trouble.
While directory services can help you turn password management into a self-service visit to a web site, some users still need to know what they can expect if they need your help.
Creating an SLO, even if your first version consists of guesses, helps you reduce performance tuning from an open-ended project to a clear set of measurable goals for a manageable project with a definite outcome.
Resource constraints
With your SLOs in hand, inventory the server, networks, storage, people, and other resources at your disposal. Now is the time to estimate whether it is possible to meet the requirements at all.
If, for example, you are expected to serve more throughput than the network can transfer, maintain high-availability with only one physical machine, store 100 GB of backups on a 50 GB partition, or provide 24/7 support all alone, no amount of tuning will fix the problem.
When checking that the resources you have at least theoretically suffice to meet your requirements, do not forget that high availability in particular requires at least two of everything to avoid single points of failure. Be sure to list the resources you expect to have, when and how long you expect to have them, and why you need them. Make note of what is missing and why.
Server hardware
DS servers are pure Java applications, making them very portable. DS servers tend to perform best on single-board, x86 systems due to low memory latency.
Storage
High-performance storage is essential for handling high-write throughput. When the database stays fully cached in memory, directory read operations do not result in disk I/O. Only writes result in disk I/O. You can further improve write performance by using solid-state disks for storage or file system cache.
|
DS directory servers are designed to work with local storage for database backends. Do not use network file systems, such as NFS, where there is no guarantee that a single process has access to files. Storage area networks (SANs) and attached storage are fine for use with DS directory servers. |
Regarding database size on disk, sustained write traffic can cause the database to grow to more than twice its initial size on disk. This is normal behavior. The size on disk does not impact the DB cache size requirements.
Linux file systems
Write barriers and journaling mode for Linux file systems help avoid directory database file corruption. They make sure writes to the file system are ordered even after a crash or power failure. Make sure these features are enabled.
Some Linux distributions permanently enable write barriers. There is no administrative action to take.
Other Linux systems leave the decision to you.
If your Linux system lets you configure write barriers and journaling mode for the file system,
refer to the options for your file system in the mount command manual page for details on enabling them.
Performance tests
Even if you do not need high availability, you still need two of everything, because your test environment needs to mimic your production environment as closely as possible.
In your test environment, set up DS servers just as you do in production. Conduct experiments to determine how to best meet your SLOs.
The following command-line tools help with basic performance testing:
-
The makeldif command generates sample data with great flexibility.
-
The
addratecommand measures add and delete throughput and response time. -
The
authratecommand measures bind throughput and response time. -
The
modratecommand measures modification throughput and response time. -
The
searchratecommand measures search throughput and response time.
All *rate commands display response time distributions measurements,
and support testing at specified levels of throughput.
For additional precision when evaluating response times, use the global configuration setting etime-resolution. To change elapsed processing time resolution from milliseconds (default) to nanoseconds:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set etime-resolution:nanoseconds \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-promptThe etime, recorded in the server access log, indicates the elapsed time to process the request.
The etime starts when the decoded operation is available to be processed by a worker thread.
Test performance with your production-ready configuration. If, however, you simply want to demonstrate top performance, take the following points into account:
-
Incorrect JVM tuning slows down server and tool performance.
Make sure the JVM is tuned for best performance. For details, refer to Java settings.
-
Unfiltered access logs record messages for each client request. Turn off full access logging.
For example, set
enabled:falsefor theJson File-Based Access Loggerlog publisher, and any other unfiltered log publishers that are enabled. -
Secure connections are recommended, and they can be costly.
Set
require-secure-authentication:falsein the password policies governing the bind entries, and bind using insecure connections.
Performance settings
Use the following suggestions when your tests show that DS performance is lacking, even though you have the right underlying network, hardware, storage, and system resources in place.
Maximum open files
DS servers must open many file descriptors when handling thousands of client connections.
Linux systems often set a limit of 1024 per user. That setting is too low to accept thousands of client connections.
Make sure the server can use at least 64K (65536) file descriptors.
For example, when running the server as user opendj on a Linux system
that uses /etc/security/limits.conf to set user level limits,
set soft and hard limits by adding these lines to the file:
opendj soft nofile 65536
opendj hard nofile 131072The example above assumes the system has enough file descriptors available overall. Check the Linux system overall maximum as follows:
$ cat /proc/sys/fs/file-max
204252Linux page caching
Default Linux virtual memory settings cause significant buildup of dirty data pages before flushing them. When the kernel finally flushes the pages to disk, the operation can exhaust the disk I/O for up to several seconds. Application operations waiting on the file system to synchronize to disk are blocked.
The default virtual memory settings can therefore cause DS server operations to block for seconds at a time. Symptoms included high outlier etimes, even for very low average etimes. For sustained high loads, such as import operations, the server has to maintain thousands of open file descriptors.
To avoid these problems, tune Linux page caching.
As a starting point for testing and tuning, set vm.dirty_background_bytes to one quarter of the disk I/O per second,
and vm.dirty_expire_centisecs to 1000 (10 seconds) using the sysctl command.
This causes the kernel to flush more often, and limits the pauses to a maximum of 250 milliseconds.
For example, if the disk I/O is 80 MB/second for writes, the following example shows an appropriate starting point.
It updates the /etc/sysctl.conf file to change the setting permanently,
and uses the sysctl -p command to reload the settings:
$ echo vm.dirty_background_bytes=20971520 | sudo tee -a /etc/sysctl.conf
[sudo] password for admin:
$ echo vm.dirty_expire_centisecs=1000 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
vm.dirty_background_bytes = 20971520
vm.dirty_expire_centisecs = 1000Be sure to test and adjust the settings for your deployment.
For additional details, refer to the Oracle documentation on
Linux Page Cache Tuning,
and the Linux sysctl command
virtual memory kernel reference.
Java settings
Default Java settings let you evaluate DS servers using limited system resources. For production systems, test and run with a tuned JVM.
|
To apply JVM settings, either:
|
-
Use the most recent supported Java environment.
Refer to the release notes section on Java for details.
-
Set the maximum heap size with
-Xmx<size>.Use at least a 2 GB heap unless your data set is small.
For additional details, refer to Cache internal nodes.
For Java 17, there is no need to set the minimum heap size. If you use Java 11, set the minimum heap size to the same value as the maximum heap size.
-
Use the default garbage collection (GC) settings, equivalent to
-XX:+UseG1GC. -
Set the maximum tenuring threshold to reduce unnecessary copying with
-XX:MaxTenuringThreshold=1.This option sets the maximum number of GC cycles an object stays in survivor spaces before it is promoted into the old generation space. The recommended setting reduces the new generation GC frequency and duration. The JVM quickly promotes long-lived objects to the old generation space, rather than letting them accumulate in new generation survivor spaces, copying them for each GC cycle.
-
(Optional) Review the following additional details for specific use cases:
-XX:+DisableExplicitGC-
When using JMX, add this option to the list of
start-ds.java-argsarguments to avoid periodic full GC events.JMX is based on RMI, which uses references to objects. By default, the JMX client and server perform a full GC periodically to clean up stale references. As a result, the default settings cause JMX to cause a full GC every hour.
Avoid using this argument with
import-ldif.offline.java-argsor when using theimport-ldifcommand. The import process uses GC to manage memory and references to memory-mapped files. -Xlog:gc=level:file-
When diagnosing JVM tuning problems, log GC messages. You can turn the option off when everything is running smoothly.
Always specify the output file for the GC log. Otherwise, the JVM logs the messages to the
opendj/logs/server.outfile, mixing them with other messages, such as stack traces from thesupportextractcommand.For example,
-Xlog:gc=info:file=/path/to/gc.loglogs informational GC messages to the file,/path/to/gc.log.For details, use the
java -Xlog:helpcommand. -XX:TieredStopAtLevel=1-
When trying to reduce startup time for short-lived client tools, such as the
ldapsearchcommand, use this setting as shown.
ForgeRock does not recommend using ZGC or huge pages at this time.
Data storage settings
By default, DS servers compress attribute descriptions and object class sets to reduce data size. This is called compact encoding.
By default, DS servers do not compress entries stored in its backend database.
If your entries hold values that compress well, such as text, you can gain space.
Set the backend property entries-compressed:true, and reimport the data from LDIF.
The DS server compresses entries before writing them to the database:
$ dsconfig \
set-backend-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--backend-name dsEvaluation \
--set entries-compressed:true \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
$ import-ldif \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--ldifFile backup.ldif \
--backendID dsEvaluation \
--includeBranch dc=example,dc=com \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pinDS directory servers do not proactively rewrite all entries after you change the settings. To force the DS server to compress all entries, you must import the data from LDIF.
LDIF import settings
By default, the temporary directory used for scratch files is opendj/import-tmp.
Use the import-ldif --tmpDirectory option to set this directory to a tmpfs file system,
such as /tmp.
If you are certain your LDIF contains only valid entries with correct syntax, you can skip schema validation.
Use the import-ldif --skipSchemaValidation option.
Database cache settings
|
By default, DS directory servers:
|
If you require fine-grained control over JE backend cache settings, you can configure the amount of memory requested for database cache per database backend:
-
Configure
db-cache-percentordb-cache-sizefor each JE backend.db-cache-percent-
Percentage of JVM memory to allocate to the database cache for the backend.
If the directory server has multiple database backends, the total percent of JVM heap used must remain less than 100 (percent), and must leave space for other uses.
Default: 50 (percent)
db-cache-size-
JVM memory to allocate to the database cache.
This is an alternative to
db-cache-percent. If you set its value larger than 0, then it takes precedence overdb-cache-percent.Default: 0 MB
-
Set the global property
je-backend-shared-cache-enabled:false. -
Restart the server for the changes to take effect.
Cache internal nodes
A JE backend has a B-tree data structure. A B-tree consists of nodes that can have children. Nodes with children are internal nodes. Nodes without children are leaf nodes.
The directory stores data in key-value pairs. Internal nodes hold the keys and can hold small values. Leaf nodes hold the values. One internal node usually holds keys to values in many leaf nodes. A B-tree has many more leaf nodes than internal nodes.
To read a value by its key, the backend traverses all internal nodes on the branch from the B-tree root to the leaf node holding the value. The closer a node is to the B-tree root, the more likely the backend must access it to get to the value. In other words, the backend accesses internal nodes more often than leaf nodes.
When a backend accesses a node, it loads the node into the DB cache. Loading a node because it wasn’t in cache is a cache miss. When you first start DS, all requests result in cache misses until the server loads active nodes.
As the DB cache fills, the backend makes space to load nodes by evicting nodes from the cache. The backend evicts leaf nodes, then least recently used internal nodes. As a last resort, the backend evicts even recently used internal nodes with changes not yet synced to storage.
The next time the backend accesses an evicted node, it must load the node from storage. Storage may mean the file system cache, or it may mean a disk. Reading from memory can be orders of magnitude faster than reading from disk. For the best DB performance, cache the nodes the DB accesses most often, which are the internal nodes.
Once DS has run for some time and active nodes are in cache, watch the cache misses for internal nodes. DS has "warmed up" and the active nodes are in the cache. The number of evicted internal nodes should remain constant. When the cache size is right, and no sudden changes occur in access patterns, the number of cache misses for internal nodes should stop growing:
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword password \
--baseDN cn=monitor \
"(objectClass=ds-monitor-backend-db)" \
ds-mon-db-cache-evict-internal-nodes-count \
ds-mon-db-cache-misses-internal-nodes
dn: ds-cfg-backend-id=dsEvaluation,cn=backends,cn=monitor
ds-mon-db-cache-evict-internal-nodes-count: <number>
ds-mon-db-cache-misses-internal-nodes: <number>If ds-mon-db-cache-evict-internal-nodes-count is greater than 0 and growing,
or ds-mon-db-cache-misses-internal-nodes continues to grow even after DS has warmed up,
DS is evicting internal nodes from the DB cache.
If you can rule out big changes in access cache patterns like large unindexed searches,
DS does not have enough space for the DB cache.
Increase the DB cache size and add more RAM to your system if necessary.
If adding RAM isn’t an option, increase the maximum heap size (-Xmx) to optimize RAM allocation.
Estimate minimum DB cache size
This section explains how to estimate the minimum DB cache size.
The examples below use a directory server with a 10 million entry
dsEvaluation backend.
The backend holds entries generated using the --set ds-evaluation/generatedUsers:10,000,000 setup option.
|
Before estimating DB cache size for your deployment:
|
After simulating realistic traffic for some time, stop the server.
Use the output from the backendstat command to estimate the required DB cache size
JE DbCacheSize tool
together to estimate the required DB cache size:
# Stop the server before using backendstat:
$ stop-ds
$ backendstat list-raw-dbs --backendId dsEvaluation
Raw DB Name Total Keys Keys Size Values Size Total Size
-----------------------------------------------------------------------------------------------------------------------------------------------
/compressed_schema/compressed_attributes 54 54 837 891
/compressed_schema/compressed_object_classes 18 18 938 956
/dc=com,dc=example/aci.presence 1 1 3 4
/dc=com,dc=example/cn.caseIgnoreMatch 10000165 139242470 47887210 187129680
/dc=com,dc=example/cn.caseIgnoreSubstringsMatch:6 858657 5106079 204936276 210042355
/dc=com,dc=example/counter.objectClass.big.objectIdentifierMatch 2 34 2 36
/dc=com,dc=example/counter.userPassword.big.passwordStorageSchemeEqualityMatch 0 0 0 0
/dc=com,dc=example/dn2id 10000181 268892913 80001448 348894361
/dc=com,dc=example/ds-certificate-fingerprint.caseIgnoreMatch 0 0 0 0
/dc=com,dc=example/ds-certificate-subject-dn.distinguishedNameMatch 1 18 3 21
/dc=com,dc=example/ds-sync-conflict.distinguishedNameMatch 0 0 0 0
/dc=com,dc=example/ds-sync-hist.changeSequenceNumberOrderingMatch 0 0 0 0
/dc=com,dc=example/entryUUID.uuidMatch 9988518 39954072 47871653 87825725
/dc=com,dc=example/givenName.caseIgnoreMatch 8614 51690 20017387 20069077
/dc=com,dc=example/givenName.caseIgnoreSubstringsMatch:6 19651 97664 48312525 48410189
/dc=com,dc=example/id2childrencount 8 26 14 40
/dc=com,dc=example/id2entry 10000181 80001448 5379599451 5459600899
/dc=com,dc=example/json.caseIgnoreJsonQueryMatch 4 56 8 64
/dc=com,dc=example/jsonToken.extensibleJsonEqualityMatch:caseIgnoreStrings:ignoreWhiteSpace:/id 2 34 4 38
/dc=com,dc=example/mail.caseIgnoreMatch 10000152 238891751 47887168 286778919
/dc=com,dc=example/mail.caseIgnoreSubstringsMatch:6 1222798 7336758 112365097 119701855
/dc=com,dc=example/member.distinguishedNameMatch 1 40 2 42
/dc=com,dc=example/oauth2Token.caseIgnoreOAuth2TokenQueryMatch 4 74 10 84
/dc=com,dc=example/objectClass.big.objectIdentifierMatch 6 156 0 156
/dc=com,dc=example/objectClass.objectIdentifierMatch 24 396 395 791
/dc=com,dc=example/referral 0 0 0 0
/dc=com,dc=example/sn.caseIgnoreMatch 13457 92943 20027045 20119988
/dc=com,dc=example/sn.caseIgnoreSubstringsMatch:6 41585 219522 73713958 73933480
/dc=com,dc=example/state 26 1335 25 1360
/dc=com,dc=example/telephoneNumber.telephoneNumberMatch 9989952 109889472 47873522 157762994
/dc=com,dc=example/telephoneNumber.telephoneNumberSubstringsMatch:6 1111110 6543210 221281590 227824800
/dc=com,dc=example/uid.caseIgnoreMatch 10000152 118889928 47887168 166777096
/dc=com,dc=example/uniqueMember.uniqueMemberMatch 10 406 21 427
/dc=com,dc=example/userPassword.big.passwordStorageSchemeEqualityMatch 0 0 0 0
Total: 34
# Calculate the sum of total keys, the average key size, and the average value size.
# Sum of total keys: 73255334
# Average key size: sum of key sizes/sum of total keys = 1015212568 / 73255334 ~= 13.86
# Average value size: sum of values sizes/sum of total keys = 6399663760 / 73255334 ~= 87.36
# Use the results rounded to the nearest integer as arguments to the DbCacheSize tool:
$ java -cp /path/to/opendj/lib/opendj.jar com.sleepycat.je.util.DbCacheSize \
-records 73255334 -key 14 -data 87
=== Environment Cache Overhead ===
3,158,773 minimum bytes
To account for JE daemon operation, record locks, HA network connections, etc,
a larger amount is needed in practice.
=== Database Cache Size ===
Number of Bytes Description
--------------- -----------
2,953,929,424 Internal nodes only
12,778,379,408 Internal nodes and leaf nodes
For further information see the DbCacheSize javadoc.The resulting recommendation for caching Internal nodes only is 2,953,929,424 bytes (~ 3 GB) in this example.
This setting for DB cache includes space for all internal nodes, including those with keys and data.
To cache all DB data, Internal nodes and leaf nodes, would require 12,778,379,408 (~13 GB).
Round up when configuring backend settings for db-cache-percent or db-cache-size.
If the system in this example has 8 GB available memory, use the default setting of db-cache-percent: 50.
(50% of 8 GB is 4 GB, which is larger than the minimum estimate.)
Database log file settings
With default settings, if the database has more than 200 files on disk, then the JE backend must start closing one log file in order to open another. This has serious impact on performance when the file cache starts to thrash.
Having the JE backend open and close log files from time to time is okay. Changing the settings is only necessary if the JE backend has to open and close the files very frequently.
A JE backend stores data on disk in append-only log files. The maximum size of each log file is configurable. A JE backend keeps a configurable maximum number of log files open, caching file handles to the log files. The relevant JE backend settings are the following:
db-log-file-max-
Maximum size of a database log file.
Default: 1 GB
db-log-filecache-size-
File handle cache size for database log files.
Default: 200
With these defaults, if the size of the database reaches 200 GB on disk (1 GB x 200 files),
the JE backend must close one log file to open another.
To avoid this situation, increase db-log-filecache-size
until the JE backend can cache file handles to all its log files.
When changing the settings, make sure the maximum number of open files is sufficient.
Log settings
Debug-level log settings trace the internal workings of DS servers, and should be used sparingly. Be careful when activating debug-level logging in high-performance deployments.
In general, leave other logs active for production environments to help troubleshoot any issues that arise.
For servers handling 100,000 operations per second or more, the access log can be a performance bottleneck. Each client request results in at least one access log message. Test whether disabling the access log improves performance in such cases.
The following command disables the JSON-based LDAP access logger:
$ dsconfig \
set-log-publisher-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--publisher-name "Json File-Based Access Logger" \
--set enabled:false \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-promptThe following command disables the HTTP access logger:
$ dsconfig \
set-log-publisher-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--publisher-name "File-Based HTTP Access Logger" \
--set enabled:false \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-promptChangelog settings
To let legacy applications get update notifications by change number, as described in Align draft change numbers, enable change number indexing.
Change number indexing requires more CPU, disk access, and storage. Enable it only when applications require change number-based browsing.