Maintenance
This guide covers recurring administrative operations.
ForgeRock® Identity Platform serves as the basis for our simple and comprehensive Identity and Access Management solution. We help our customers deepen their relationships with their customers, and improve the productivity and connectivity of their employees and partners. For more information about ForgeRock and about the platform, see https://www.forgerock.com.
Maintenance tools
Server commands
-
Add DS server command-line tools to your PATH:
-
Bash
-
PowerShell
$ export PATH=/path/to/opendj/bin:$\{PATH}
PS C:\path\to> $env:PATH += ";C:\path\to\opendj\bat"
-
-
For reference information, use the
--help
option with any DS tool. -
All commands call Java programs. This means every command starts a JVM, so it takes longer to start than a native binary.
-
The DS
bash-completion
command generates a completion script for the Bash shell that makes it easier to write other DS commands.The completion script depends on support for
bash-completion
, which is not included by default on macOS.To set up Bash completion for DS commands, source the output of the script:
-
Bash 4
-
Bash 3.2 macOS
source <(/path/to/opendj/bin/bash-completion)
# First, install bash-completion support. # Next: eval "$( /path/to/opendj/bin/bash-completion )"
You can make completion available in any new interactive shell by adding it to your
~/.bash_profile
file, or~/.bashrc
file if it is loaded by the new shell. -
DS running on… | DS installed from… | Default path to tools… |
---|---|---|
Linux distributions |
.zip |
|
Linux distributions |
.deb, .rpm |
|
Microsoft Windows |
.zip |
|
The installation and upgrade tools, setup
,
and upgrade
, are found in the parent directory of the other tools.
These tools are not used for everyday administration.
Commands | Constraints |
---|---|
|
When the server is offline, or when running commands in offline mode, these commands can modify server files. They must, therefore, access server files as a user who has the same filesystem permissions as the user who installs and runs the server. For most systems, the simplest way to achieve this is to run the command as the same user who installs and runs the server. When following best practices for auditing and separation of duty, provision administrative and server user accounts with compatible group or access control list permissions. |
|
These commands must be used with the local DS server in the same installation as the tools. These commands are not useful with non-DS servers. |
|
These commands must be used with DS servers having the same version as the command. These commands are not useful with non-DS servers. |
|
This command depends on template files.
The template files can make use of configuration files installed with DS servers
under The LDIF output can be used with any directory server. |
|
These commands can be used independently of DS servers, and are not tied to a specific version. |
Command(1) | Description |
---|---|
Measure add and delete throughput and response time. |
|
Measure bind throughput and response time. |
|
Debug databases for pluggable backends. |
|
Encode and decode data in base64 format. Base64-encoding represents binary data in ASCII, and can be used to encode character strings in LDIF, for example. |
|
|
Generate a completion script for use with Bash shell.
Requires |
Debug file-based changelog databases. |
|
|
Generate a This lets you register and manage DS servers as services on Linux systems. |
Back up or restore directory data. |
|
Generate a deployment ID, a shared master key, a private CA certificate based on a deployment ID and password, or a key pair with the certificate signed by the private CA. |
|
The To edit the configuration when the server is not running, use the Some advanced properties are not visible by default when you run the When you pass connection information, subcommands, and additional options to You can prepare Alternatively, you can read commands from standard input with the |
|
Manage data replication between directory servers to keep their contents in sync. |
|
Encode a plaintext password according to one of the available storage schemes. |
|
Export directory data to LDIF, the standard, portable, text-based representation of directory content. |
|
Load LDIF content into the directory, which overwrites existing data. It cannot be used to append data to the backend database. |
|
Compare the attribute values you specify with those stored on entries in the directory. |
|
Delete one entry or an entire branch of subordinate entries in the directory. |
|
Modify the specified attribute values for the specified entries. |
|
Modify user passwords. |
|
Search a branch of directory data for entries that match the LDAP filter you specify. |
|
Display differences between two LDIF files. The output is LDIF. |
|
Similar to the |
|
Similar to the |
|
Generate directory data in LDIF based on templates that define how the data should appear. The |
|
Lock and unlock user accounts, and view and manipulate password policy state information. |
|
View information about tasks scheduled to run in the server, and cancel specified tasks. |
|
Measure modification throughput and response time. |
|
Rebuild an index stored in an indexed backend. |
|
Measure search throughput and response time. |
|
Configure a setup profile after initial installation. |
|
Start one DS server. |
|
Display information about the server. |
|
Stop one DS server. |
|
Collect troubleshooting information for technical support purposes. |
|
Verify that an index stored in an indexed backend is not corrupt. |
|
|
Register and manage one DS server as a Windows service. |
(1) Linux names for the commands. Equivalent Windows commands have .bat extensions.
Trusted certificates
When a client tool initiates a secure connection to a server, the server presents its digital certificate.
The tool must decide whether it does trust the server certificate and continues to negotiate a secure connection, or doesn’t trust the server certificate and drops the connection. To trust the server certificate, the tool’s truststore must contain the trusted certificate. The trusted certificate is a CA certificate, or the self-signed server certificate.
The following table explains how the tools locate the truststore.
Truststore Option | Truststore Used |
---|---|
None |
The default truststore,
|
|
DS only uses the specified truststore. The <Type> in the option name reflects the trust store type. The tool fails with an error if it can’t trust the server certificate. |
Default settings
You can set defaults in the ~/.opendj/tools.properties
file, as in the following example:
hostname=localhost
port=4444
bindDN=editable:dsAdminDN["uid=admin"]
bindPassword\:file=/path/to/.pwd
useSsl=true
trustAll=true
When you use an option with a colon, such as bindPassword:file
,
escape the colon with a backslash (\:
) in the properties file.
The file location on Windows is %UserProfile%\.opendj\tools.properties
.
Server processes
Start a server
-
Start the server in the background:
$ start-ds
Alternatively, specify the
--no-detach
option to start the server in the foreground. -
(Linux) If the DS server was installed from a .deb or .rpm package, then service management scripts were created at setup time:
centos# service opendj start Starting opendj (via systemctl): [ OK ]
ubuntu$ sudo service opendj start $Starting opendj: > SUCCESS.
-
(Linux) Use a
systemd
service to manage DS.# Create the service. # Unless you run DS as root, # set --userName to the user who installed the server: $ sudo create-rc-script \ --systemdService /etc/systemd/system/opendj.service \ --userName opendj # Register the service you created: $ sudo systemctl daemon-reload
Manage the service with
systemctl
.$ sudo systemctl start opendj
-
(Windows) Register the DS server as a Windows service:
C:\path\to\opendj\bat> windows-service.bat --enableService
Manage the service with Windows-native administration tools.
Stop a server
Although DS servers are designed to recover from failure and disorderly shutdown, it is safer to shut the server down cleanly, because a clean shutdown reduces startup delays. During startup, the server attempts to recover database backend state. Clean shutdown prevents situations where the server cannot recover automatically.
Clean server retirement
-
Before shutting down the system where the server is running, and before detaching any storage used for directory data, cleanly stop the server using one of the following techniques:
-
Use the
stop-ds
command:$ stop-ds
-
(Linux) If the DS server was installed from a .deb or .rpm package, then service management scripts were created at setup time:
centos# service opendj stop Stopping opendj (via systemctl): [ OK ]
ubuntu$ sudo service opendj stop $Stopping opendj: ... > SUCCESS.
-
(Linux) Use a
systemd
service to manage DS.# Create the service. # Unless you run DS as root, # set --userName to the user who installed the server: $ sudo create-rc-script \ --systemdService /etc/systemd/system/opendj.service \ --userName opendj # Register the service you created: $ sudo systemctl daemon-reload
Manage the service with
systemctl
.$ sudo systemctl stop opendj
-
(Windows) Register the DS server once as a Windows service:
C:\path\to\opendj\bat> windows-service.bat --enableService
Manage the service with Windows-native administration tools.
Do not intentionally kill the DS server process unless the server is completely unresponsive. When stopping cleanly, the server writes state information to database backends, and releases locks that it holds on database files.
-
Restart a server
-
Use the
stop-ds
command:$ stop-ds --restart
-
(Linux) If the DS server was installed from a .deb or .rpm package, then service management scripts were created at setup time:
centos# service opendj restart Restarting opendj (via systemctl): [ OK ]
ubuntu$ sudo service opendj restart $Stopping opendj: ... > SUCCESS. $Starting opendj: > SUCCESS.
-
(Linux) Use a
systemd
service to manage DS.# Create the service. # Unless you run DS as root, # set --userName to the user who installed the server: $ sudo create-rc-script \ --systemdService /etc/systemd/system/opendj.service \ --userName opendj # Register the service you created: $ sudo systemctl daemon-reload
Manage the service with
systemctl
.$ sudo systemctl restart opendj
-
(Windows) Register the DS server once as a Windows service:
C:\path\to\opendj\bat> windows-service.bat --enableService
Manage the service with Windows-native administration tools.
Server tasks
The following server administration commands can be run in online and offline mode. They invoke data-intensive operations, and so potentially take a long time to complete. The links below are to the reference documentation for each command:
When you run these commands in online mode, they run as tasks on the server.
Server tasks are scheduled operations that can run one or more times as long as the server is up.
For example, you can schedule the dsbackup
and export-ldif
commands to run recurrently
in order to back up server data on a regular basis.
You schedule a task as a directory administrator, sending the request to the administration port. You can therefore schedule a task on a remote server if you choose. When you schedule a task on a server, the command returns immediately, yet the task can start later, and might run for a long time before it completes. You can access tasks by using the manage-tasks command.
Although you can schedule a server task on a remote server, the data for the task must be accessible to the server locally. For example, when you schedule a backup task on a remote server, that server writes backup files to a file system on the remote server. Similarly, when you schedule a restore task on a remote server, that server restores backup files from a file system on the remote server.
The reference documentation describes the available options for each command:
-
Configure email notification for success and failure
-
Define alternatives on failure
-
Start tasks immediately (
--start 0
) -
Schedule tasks to start at any time in the future
Server recovery
DS servers can restart after a crash or after the server process is killed abruptly. After disorderly shutdown, the DS server must recover its database backends. Generally, DS servers return to service quickly.
Database recovery messages are found in the database log file, such as /path/to/opendj/db/userData/dj.log
.
The following example shows two example messages from the recovery log. The first message is written at the beginning of the recovery process. The second message is written at the end of the process:
[/path/to/opendj/db/userData]Recovery underway, found end of log
...
[/path/to/opendj/db/userData]Recovery finished: Recovery Info ...
The JVM’s heap-based database cache is lost when the server stops or crashes. The cache must therefore be reconstructed from the directory database files. Database files might still be in the filesystem cache on restart, but rebuilding the JVM’s heap-based database cache takes time. DS servers start accepting client requests before this process is complete.
Backup and restore
|
How backup works
DS directory servers store data in backends. The amount of data in a backend varies depending on your deployment. It can range from very small to very large. A JE backend can hold billions of LDAP entries, for example.
Backup process
A JE backend stores data on disk using append-only log files with names like number.jdb
.
The JE backend writes updates to the highest-numbered log file.
The log files grow until they reach a specified size (default: 1 GB).
When the current log file reaches the specified size, the JE backend creates a new log file.
To avoid an endless increase in database size on disk, JE backends clean their log files in the background. A cleaner thread copies active records to new log files. Log files that no longer contain active records are deleted.
The DS backup process takes advantage of this log file structure. Together, a set of log files represents a backend at a point in time. The backup process essentially copies the log files to the backup directory. DS also protects the data and adds metadata to keep track of the log files it needs to restore a JE backend to the state it had when the backup task completed.
Cumulative backups
DS backups are cumulative in nature. Backups reuse the JE files that did not change since the last backup operation. They only copy the JE files the backend created or changed. Files that did not change are shared between backups.
A set of backup files is fully standalone.
Purge old backups
Backup tasks keep JE files until you purge them.
The backup purge operation prevents an endless increase in the size of the backup folder on disk. The purge operation does not happen automatically; you choose to run it. When you run a purge operation, it removes the files for old or selected backups. The purge does not impact the integrity of the backups DS keeps. It only removes log files that do not belong to any remaining backups.
Back up
When you set up a directory server, the process creates a /path/to/opendj/bak/
directory.
You can use this for backups if you have enough local disk space, and when developing or testing backup processes.
In deployment, store backups remotely to avoid losing your data and backups in the same crash.
Back up data (server task)
When you schedule a backup as a server task, the DS server manages task completion. The server must be running when you schedule the task, and when the task runs:
-
Schedule the task on a running server, binding as a user with the
backend-backup
administrative privilege.The following example schedules an immediate backup task for the
dsEvaluation
backend:$ dsbackup \ create \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --backupLocation bak \ --backendName dsEvaluation
To back up all backends, omit the
--backendName
option.To back up more than one backend, specify the
--backendName
option multiple times.For details, refer to dsbackup.
Back up data (scheduled task)
When you schedule a backup as a server task, the DS server manages task completion. The server must be running when you schedule the task, and when the task runs:
-
Schedule backups using the
crontab
format with the--recurringTask
option.The following example schedules nightly online backup of all user data at 2 AM, notifying
diradmin@example.com
when finished, or on error:$ dsbackup \ create \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --backupLocation bak \ --recurringTask "00 02 * * *" \ --description "Nightly backup at 2 AM" \ --taskId NightlyBackup \ --completionNotify diradmin@example.com \ --errorNotify diradmin@example.com
For details, refer to dsbackup.
Back up data (external command)
When you back up data without contacting the server,
the dsbackup create
command runs as an external command, independent of the server process.
It backs up the data whether the server is running or not.
When you back up LDIF-based backends with this method, the command does not lock the files.
To avoid corrupting the backup files, do not run the This applies to LDIF backends, schema files, and the task backend, for example. |
Use this method to schedule backup with a third-party tool, such as the cron
command:
-
Back up data without contacting the server process, and use the
--offline
option.The following example backs up the
dsEvaluation
backend immediately:$ dsbackup \ create \ --offline \ --backupLocation bak \ --backendName dsEvaluation
To back up all backends, omit the
--backendName
option.To back up more than one backend, specify the
--backendName
option multiple times.For details, refer to dsbackup.
Back up configuration files
When you back up directory data using the dsbackup
command, you do not back up server configuration files.
The server stores configuration files under the /path/to/opendj/config/
directory.
The server records snapshots of its configuration under the /path/to/opendj/var/
directory.
You can use snapshots to recover from misconfiguration performed with the dsconfig
command.
Snapshots only reflect the main configuration file, config.ldif
.
-
Stop the server:
$ stop-ds
-
Back up the configuration files:
$ tar -zcvf backup-config-$(date +%s).tar.gz config
By default, this backup includes the server keystore, so store it securely.
-
Start the server:
$ start-ds
Back up using snapshots
ForgeRock recommends using the dsbackup
command when possible for backup and restore operations.
You can use snapshot technology as an alternative to the dsbackup
command,
but you must be careful how you use it.
While DS directory servers are running, database backend cleanup operations write data even when there are no pending client or replication operations. An ongoing file system backup operation may record database log files that are not in sync with each other.
Successful recovery after restore is only guaranteed under certain conditions.
The snapshots must:
-
Be atomic, capturing the state of all files at exactly the same time.
If you are not sure that the snapshot technology is atomic, do not use it. Use the
dsbackup
command instead. -
Capture the state of all data (
db/
) and (changelogDb/
) changelog files together.When using a file system-level snapshot feature, for example, keep at least all data and changelog files on the same file system. This is the case in a default server setup.
-
Be paired with a specific server configuration.
A snapshot of all files includes configuration files that may be specific to one DS server, and cannot be restored safely on another DS server with a different configuration. If you restore all system files, this principle applies to system configuration as well.
For details on making DS configuration files as generic as possible, refer to Property value substitution.
If snapshots in your deployment do not meet these criteria, you must stop the DS server before taking the snapshot. You must also take care not to restore incompatible configuration files.
Restore
After you restore a replicated backend, replication brings it up to date with changes newer than the backup.
Replication uses internal change log records to determine which changes to apply.
This process happens even if you only have a single server that you configured for replication at setup time
(by setting the replication port with the Replication purges internal change log records, however, to prevent the change log from growing indefinitely. Replication can only bring the backend up to date if the change log still includes the last change backed up. For this reason, when you restore a replicated backend from backup, the backup must be newer than the last purge of the replication change log (default: 3 days). If no backups are newer than the replication purge delay, do not restore from a backup. Initialize the replica instead, without using a backup. For details, refer to Manual initialization. |
Restore data (server task)
-
Verify the backup you intend to restore.
The following example verifies the most recent backup of the
dsEvaluation
backend:$ dsbackup \ list \ --backupLocation bak \ --backendName dsEvaluation \ --last \ --verify
-
Schedule the restore operation as a task, binding as a user with the
backend-restore
administrative privilege.The following example schedules an immediate restore task for the
dsEvaluation
backend:$ dsbackup \ restore \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --backupLocation bak \ --backendName dsEvaluation
To restore the latest backups of more than one backend, specify the
--backendName
option multiple times.To restore a specific backup, specify the
--backupId
option. To restore multiple specific backups of different backends, specify the--backupId
option multiple times.To list backup information without performing verification, use the
dsbackup list
command without the--verify
option. The output includes backup IDs for use with the--backupId
option.For details, refer to dsbackup.
Restore data (external command)
-
Stop the server if it is running:
$ stop-ds --quiet
-
Verify the backup you intend to restore.
The following example verifies the most recent backup of the
dsEvaluation
backend:$ dsbackup \ list \ --backupLocation bak \ --backendName dsEvaluation \ --last \ --verify
-
Restore using the
--offline
option.The following example restores the
dsEvaluation
backend:$ dsbackup \ restore \ --offline \ --backupLocation bak \ --backendName dsEvaluation
To restore the latest backups of more than one backend, specify the
--backendName
option multiple times.To restore a specific backup, specify the
--backupId
option. To restore multiple specific backups of different backends, specify the--backupId
option multiple times.To list backup information without performing verification, use the
dsbackup list
command without the--verify
option. The output includes backup IDs for use with the--backupId
option.For details, refer to dsbackup.
-
Start the server:
$ start-ds --quiet
Restore configuration files
-
Stop the server:
$ stop-ds --quiet
-
Restore the configuration files from the backup, overwriting existing files:
$ tar -zxvf backup-config-<date>.tar.gz
-
Start the server:
$ start-ds --quiet
Restore from a snapshot
ForgeRock recommends using the dsbackup
command when possible for backup and restore operations.
You can use snapshot technology as an alternative to the dsbackup
command,
but you must be careful how you use it.
For details, refer to Back up using snapshots.
Take the following points into account before restoring a snapshot:
-
When you restore files for a replicated backend, the snapshot must be newer than the last purge of the replication change log (default: 3 days).
-
Stop the DS server before you restore the files.
-
The DS configuration files in the snapshot must match the configuration where you restore the snapshot.
If the configuration uses expressions, define their values for the current server before starting DS.
-
When using snapshot files to initialize replication, only restore the data (
db/
) files for the target backend.Depending on the snapshot technology, you might need to restore the files separately, and then move only the target backend files from the restored snapshot.
-
When using snapshot files to restore replicated data to a known state, stop all affected servers before you restore.
Purge old files
Periodically purge old backup files with the dsbackup purge
command.
The following example removes all backup files older than the default replication purge delay:
$ dsbackup \
purge \
--offline \
--backupLocation bak \
--olderThan 3d
This example runs the external command without contacting the server process. You can also purge backups by ID, or by backend name, and you can specify the number of backups to keep. For details, refer to dsbackup.
To purge files as a server task, use the task options, such as --recurringTask
.
The user must have the backend-backup
administrative privilege to schedule a purge task.
Cloud storage
You can stream backup files to cloud storage, and restore them directly from cloud storage.
The implementation supports these providers:
-
Amazon AWS S3
-
Azure Cloud Storage
-
Google Cloud Storage
Follow these steps to store backup files in the cloud:
-
If you upgraded in place from DS 6.5 or earlier, activate cloud storage for backup.
-
Get a storage account and space from the cloud provider where the server can store backup files.
This storage space is referred to below as cloud-bak.
-
Get credentials from the cloud provider.
The DS server backing up files must have read, write, and delete access. For information about granting access, refer to the access control documentation for your provider.
If you are not yet familiar with cloud storage, refer to the documentation from your provider for help. The following table provides links to the documentation for supported providers:
Provider Hints Amazon AWS S3
For details on setting up S3 and working with S3 buckets, refer to the Amazon Web Services documentation on Getting started with Amazon Simple Storage Service.
Azure Cloud Storage
DS authenticates to Azure with an Azure storage account. For details, refer to the Microsoft documentation on how to Create an Azure Storage account, or to Create a BlockBlobStorage account.
Google Cloud Storage
DS authenticates to Google Cloud with a service account. For details, refer to the Google documentation on Getting Started with Authentication.
For details about creating and managing storage buckets, refer to the Google How-To documentation on Creating buckets, and Working with buckets.
-
Set environment variables for the credentials:
Provider Environment Variable(s) Amazon AWS S3
export AWS_ACCESS_KEY_ID=aws-access-key
export AWS_SECRET_ACCESS_KEY=aws-secret-key
When using temporary credentials, also export the session token:
export AWS_SESSION_TOKEN=aws-session-token
Azure Cloud Storage
export AZURE_ACCOUNT_NAME=azure-account-name
export AZURE_ACCOUNT_KEY=azure-account-key
Google Cloud Storage
export GOOGLE_CREDENTIALS=/path/to/gcp-credentials.json
(optional) -
Restart the DS server so that it reads the environment variables you set:
$ stop-ds --restart
-
Run
dsbackup
commands with all required provider-specific options.The options in the following table use the providers' default storage endpoints:
Provider Required Options Amazon AWS S3
--storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \ --storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \ --backupLocation s3://cloud-bak # When using temporary credentials, also use the session token: --storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \ --storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \ --storageProperty s3.sessionToken.env.var:AWS_SESSION_TOKEN \ --backupLocation s3://cloud-bak
Azure Cloud Storage
--storageProperty az.accountName.env.var:AZURE_ACCOUNT_NAME \ --storageProperty az.accountKey.env.var:AZURE_ACCOUNT_KEY \ --backupLocation az://cloud-bak
Google Cloud Storage
--storageProperty gs.credentials.path:/path/to/gcp-credentials.json \ --backupLocation gs://cloud-bak
or
--storageProperty gs.credentials.env.var:GOOGLE_CREDENTIALS \ --backupLocation gs://cloud-bak
In production environments, also set the cloud storage endpoint.
Cloud storage requires working space in the local system temporary directory. Some cloud storage providers require sending the content length with each file.
To send the correct content length, the
dsbackup
command writes each prepared backup file to the system temporary directory before upload. It deletes each file after successful upload.
Cloud storage endpoint
Backup to cloud storage can use a default endpoint, which can simplify evaluation and testing.
Control where your backup files go. Add one of the following options:
-
--storage-property endpoint:endpoint-url
-
--storage-property endpoint.env.var:environment-variable-for-endpoint-url
The endpoint-url depends on your provider.
Refer to their documentation for details.
For Azure cloud storage, the endpoint-url starts with the account name.
Examples include https://azure-account-name.blob.core.windows.net
,
https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net
,
and https://${AZURE_ACCOUNT_NAME}.some.private.azure.endpoint
.
Cloud storage samples demonstrate how to use the setting.
Cloud storage samples
Click the samples for your storage provider to expand the section and display the commands:
AWS samples
#
# API keys created through the AWS API gateway console:
#
export AWS_ACCESS_KEY_ID=aws-access-key-id
export AWS_SECRET_ACCESS_KEY=aws-secret-key
# When using temporary credentials:
# export AWS_SESSION_TOKEN=aws-session-token
# These samples use the following S3 bucket, and a non-default endpoint:
# S3 bucket: s3://ds-test-backup
# S3 endpoint: https://s3.us-east-1.amazonaws.com
#
# When using temporary credentials, also add
# the AWS session token storage property option to each of the commands:
# --storageProperty s3.sessionToken.env.var:AWS_SESSION_TOKEN
# Back up the dsEvaluation backend offline:
dsbackup create --backendName dsEvaluation --offline \
--backupLocation s3://ds-test-backup \
--storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \
--storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \
--storageProperty endpoint:https://s3.us-east-1.amazonaws.com
# List and verify the latest backup files for each backend at this location:
dsbackup list --verify --last \
--backupLocation s3://ds-test-backup \
--storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \
--storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \
--storageProperty endpoint:https://s3.us-east-1.amazonaws.com
# Restore dsEvaluation from backup offline:
dsbackup restore --backendName dsEvaluation --offline \
--backupLocation s3://ds-test-backup \
--storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \
--storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \
--storageProperty endpoint:https://s3.us-east-1.amazonaws.com
# Purge all dsEvaluation backup files:
dsbackup purge --backendName dsEvaluation --keepCount 0 --offline \
--backupLocation s3://ds-test-backup \
--storageProperty s3.keyId.env.var:AWS_ACCESS_KEY_ID \
--storageProperty s3.secret.env.var:AWS_SECRET_ACCESS_KEY \
--storageProperty endpoint:https://s3.us-east-1.amazonaws.com
Azure samples
#
# Credentials for Azure storage, where the Azure account is found in key1 in the Azure console:
#
export AZURE_ACCOUNT_NAME=azure-account-name
export AZURE_ACCOUNT_KEY=azure-account-key
# These samples use the following Azure storage, and a non-default endpoint:
# Azure storage: az://ds-test-backup/test1
# Azure endpoint: https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net
# Back up the dsEvaluation backend offline:
dsbackup create --backendName dsEvaluation --offline \
--backupLocation az://ds-test-backup/test1 \
--storageProperty az.accountName.env.var:AZURE_ACCOUNT_NAME \
--storageProperty az.accountKey.env.var:AZURE_ACCOUNT_KEY \
--storageProperty "endpoint:https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net"
# List and verify the latest backup files for each backend at this location:
dsbackup list --verify --last \
--backupLocation az://ds-test-backup/test1 \
--storageProperty az.accountName.env.var:AZURE_ACCOUNT_NAME \
--storageProperty az.accountKey.env.var:AZURE_ACCOUNT_KEY \
--storageProperty "endpoint:https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net"
# Restore dsEvaluation from backup offline:
dsbackup restore --backendName dsEvaluation --offline \
--backupLocation az://ds-test-backup/test1 \
--storageProperty az.accountName.env.var:AZURE_ACCOUNT_NAME \
--storageProperty az.accountKey.env.var:AZURE_ACCOUNT_KEY \
--storageProperty "endpoint:https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net"
# Purge all dsEvaluation backup files:
dsbackup purge --backendName dsEvaluation --keepCount 0 --offline \
--backupLocation az://ds-test-backup/test1 \
--storageProperty az.accountName.env.var:AZURE_ACCOUNT_NAME \
--storageProperty az.accountKey.env.var:AZURE_ACCOUNT_KEY \
--storageProperty "endpoint:https://${AZURE_ACCOUNT_NAME}.blob.core.windows.net"
Google cloud samples
#
# Credentials generated with and download from the Google cloud console:
#
export GOOGLE_CREDENTIALS=/path/to/gcp-credentials.json
# These samples use the following cloud storage, and endpoint:
# Google storage: gs://ds-test-backup/test1
# Google endpoint: https://www.googleapis.com
# Back up the dsEvaluation backend offline:
dsbackup create --backendName dsEvaluation --offline \
--backupLocation gs://ds-test-backup/test1 \
--storageProperty gs.credentials.env.var:GOOGLE_CREDENTIALS \
--storageProperty endpoint:https://www.googleapis.com
# List and verify the latest backup files for each backend at this location:
dsbackup list --verify --last \
--backupLocation gs://ds-test-backup/test1 \
--storageProperty gs.credentials.env.var:GOOGLE_CREDENTIALS \
--storageProperty endpoint:https://www.googleapis.com
# Restore dsEvaluation from backup offline:
dsbackup restore --backendName dsEvaluation --offline \
--backupLocation gs://ds-test-backup/test1 \
--storageProperty gs.credentials.env.var:GOOGLE_CREDENTIALS \
--storageProperty endpoint:https://www.googleapis.com
# Purge all dsEvaluation backup files:
dsbackup purge --backendName dsEvaluation --keepCount 0 --offline \
--backupLocation gs://ds-test-backup/test1 \
--storageProperty gs.credentials.env.var:GOOGLE_CREDENTIALS \
--storageProperty endpoint:https://www.googleapis.com
Efficiently store backup files
DS backups are collections of files in a backup directory. To restore from backup, DS requires a coherent collection of backup files.
You can use the dsbackup command to purge stale backup files from a backup directory. When you purge stale backup files, the command leaves a coherent collection of files you can use to restore data.
You should also store copies of backup files remotely to guard against the loss of data in a disaster.
Remote storage
Perform the following steps to store copies of backup files remotely in an efficient way. These steps address backup of directory data, which is potentially very large, not backup of configuration data, which is almost always small:
-
Choose a local directory or local network directory to hold backup files.
Alternatively, you can back up to cloud storage.
-
Schedule a regular backup task to back up files to the directory you chose.
Make sure that the backup task runs more often than the replication purge delay. For example, schedule the backup task to run every three hours for a default purge delay of three days. Each time the task runs, it backs up only new directory backend files.
For details, refer to the steps for backing up directory data.
-
Store copies of the local backup files at a remote location for safekeeping:
-
Purge old files in the local backup directory.
As described in How backup works, DS backups are cumulative in nature; DS reuses common data that has not changed from previous backup operations when backing up data again. The set of backup files is fully standalone.
The purge removes stale files without impacting the integrity of newer backups, reducing the volume of backup files to store when you copy files remotely.
-
Regularly copy the backup directory and all the files it holds to a remote location.
For example, copy all local backup files every day to a remote directory called
bak-date
:$ ssh user@remote-storage mkdir /path/to/bak-date $ scp -R /path/to/bak/* user@remote-storage:/path/to/bak-date/
-
-
Remove old
bak-date
directories from remote storage in accordance with the backup policy for the deployment.
Restore from remote backup
For each DS directory server to restore:
-
Install DS using the same cryptographic keys and deployment ID.
Backup files are protected using keys derived from the DS deployment ID and password. You must use the same ones when recovering from a disaster.
-
Restore directory data from the latest remote backup folder.
After restoring all directory servers, validate that the restore procedure was a success.
Disaster recovery
Directory services are critical to authentication, session management, authorization, and more. When directory services are broken, quick recovery is a must.
In DS directory services, a disaster is a serious data problem affecting the entire replication topology. Replication can’t help you recover from a disaster because it replays data changes everywhere.
Disaster recovery comes with a service interruption, the loss of recent changes, and a reset for replication. It is rational in the event of a real disaster. It’s unnecessary to follow the disaster recovery procedure for a hardware failure or a server that’s been offline too long and needs reinitialization. Even if you lose most of your DS servers, you can still rebuild the service without interruption or data loss.
For disaster recovery to be quick, you must prepare in advance. Don’t go to production until you have successfully tested your disaster recovery procedures. |
The following example helps prepare to recover from a disaster. It shows the following tasks:
-
Back up a DS directory service.
-
Restore the service to a known state.
-
Validate the procedure.
Tasks
The following tasks demonstrate a disaster recovery procedure on a single computer two replicated DS servers set up for evaluation.
In deployment, the procedure involves multiple computers, but the order and content of the tasks remain the same. Before you perform the procedure in production, make sure you have copies of the following:
-
The deployment description, documentation, plans, runbooks, and scripts.
-
The system configuration and software, including the Java installation.
-
The DS software and any customizations, plugins, or extensions.
-
A recent backup of any external secrets required, such as an HSM or a CA key.
-
A recent backup of each server’s configuration files, matching the production configuration.
-
The deployment ID and password.
This procedure applies to DS versions providing the For deployments with any earlier DS servers that don’t provide the command, you can’t use this procedure. Instead, refer to How do I perform disaster recovery steps in DS (All versions)? |
Disaster recovery has these characteristics:
-
You perform disaster recovery on a stopped server, one server at a time.
-
Disaster recovery is per base DN, like replication.
-
On each server you recover, you use the same disaster recovery ID, a unique identifier for this recovery.
To minimize the service interruption, this example recovers the servers one by one. It is also possible to perform disaster recovery in parallel by stopping and starting all servers together.
Task 1: Back up directory data
Back up data while the directory service is running smoothly. For additional details, refer to Backup and restore.
-
Back up the directory data.
The following command backs up directory data created for evaluation:
$ /path/to/opendj/bin/dsbackup \ create \ --start 0 \ --backupLocation /path/to/opendj/bak \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin
The command returns, and the DS server runs the backup task in the background.
When adapting the recovery process for deployment, schedule a backup task to run regularly for each database backend.
-
Check the backup task finishes successfully:
$ /path/to/opendj/bin/manage-tasks \ --summary \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
The status of the backup task is "Completed successfully" when it is done.
Recovery from disaster means stopping the directory service and losing the latest changes. The more recent the backup, the fewer changes you lose during recovery. Backup operations are cumulative, so you can schedule them regularly without using too much disk space as long as you purge outdated backup files. As you script your disaster recovery procedures for deployment, schedule a recurring backup task to have safe, current, and complete backup files for each backend.
Task 2: Recover from a disaster
This task restores the directory data from backup files created before the disaster. Adapt this procedure as necessary if you have multiple directory backends to recover.
All changes since the last backup operation are lost. |
Subtasks:
Prepare for recovery
-
If you have lost DS servers, replace them with servers configured as before the disaster.
In this example, no servers were lost. Reuse the existing servers.
-
On each replica, prevent applications from making changes to the backend for the affected base DN. Changes made during recovery would be lost or could not be replicated:
$ /path/to/opendj/bin/dsconfig \ set-backend-prop \ --backend-name dsEvaluation \ --set writability-mode:internal-only \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt $ /path/to/replica/bin/dsconfig \ set-backend-prop \ --backend-name dsEvaluation \ --set writability-mode:internal-only \ --hostname localhost \ --port 14444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
In this example, the first server’s administrative port is
4444
. The second server’s administrative port is14444
.
Recover the first directory server
DS uses the disaster recovery ID to set the generation ID, an internal, shorthand form of the initial replication state. Replication only works when the data for the base DN share the same generation ID on each server. There are two approaches to using the
Don’t mix the two approaches in the same disaster recovery procedure. Use the generated recovery ID or the recovery ID of your choice, but do not use both. |
This process generates the disaster recovery ID to use when recovering the other servers.
-
Stop the directory server you use to start the recovery process:
$ /path/to/opendj/bin/stop-ds
-
Restore the affected data on this directory server:
$ /path/to/opendj/bin/dsbackup \ restore \ --offline \ --backendName dsEvaluation \ --backupLocation /path/to/opendj/bak
Changes to the affected data that happened after the backup are lost. Use the most recent backup files prior to the disaster.
This approach to restoring data works in deployments with the same DS server version. When all DS servers share the same DS version, you can restore all the DS directory servers from the same backup data.
Backup archives are not guaranteed to be compatible across major and minor server releases. Restore backups only on directory servers of the same major or minor version.
-
Run the command to begin the disaster recovery process.
When this command completes successfully, it displays the disaster recovery ID:
$ /path/to/opendj/bin/dsrepl \ disaster-recovery \ --baseDn dc=example,dc=com \ --generate-recovery-id \ --no-prompt Disaster recovery id: <generatedId>
Record the <generatedId>. You will use it to recover all other servers.
-
Start the recovered server:
$ /path/to/opendj/bin/start-ds
-
Test the data you restored is what you expect.
-
Start backing up the recovered directory data.
As explained in New backup after recovery, you can no longer rely on pre-recovery backup data after disaster recovery.
-
Allow external applications to make changes to directory data again:
$ /path/to/opendj/bin/dsconfig \ set-backend-prop \ --backend-name dsEvaluation \ --set writability-mode:enabled \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
You have recovered this replica and begun to bring the service back online. To enable replication with other servers to resume, recover the remaining servers.
Recover remaining servers
Make sure you have a disaster recovery ID. Use the same ID for all DS servers in this recovery procedure:
|
You can perform this procedure in parallel on all remaining servers or on one server at a time. For each server:
-
Stop the server:
$ /path/to/replica/bin/stop-ds
-
Unless the server is a standalone replication server, restore the affected data:
$ /path/to/replica/bin/dsbackup \ restore \ --offline \ --backendName dsEvaluation \ --backupLocation /path/to/opendj/bak
-
Run the recovery command.
The following command uses a generated ID. It verifies this server’s data matches the first server you recovered:
$ export DR_ID=<generatedId> $ /path/to/replica/bin/dsrepl \ disaster-recovery \ --baseDn dc=example,dc=com \ --generated-id ${DR_ID} \ --no-prompt
If the recovery ID is a unique ID of your choosing, use
dsrepl disaster-recovery --baseDn <base-dn> --user-generated-id <recoveryId>
instead. This alternative doesn’t verify the data on each replica match and won’t work if the replication topology includes one or more standalone replication servers. -
Start the recovered server:
$ /path/to/replica/bin/start-ds
-
If this is a directory server, test the data you restored is what you expect.
-
If this is a directory server, allow external applications to make changes to directory data again:
$ /path/to/replica/bin/dsconfig \ set-backend-prop \ --backend-name dsEvaluation \ --set writability-mode:enabled \ --hostname localhost \ --port 14444 \ --bindDN uid=admin \ --bindPassword password \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
After completing these steps for all servers, you have restored the directory service and recovered from the disaster.
Validation
After recovering from the disaster, validate replication works as expected. Use the following steps as a simple guide.
-
Modify an entry on one replica.
The following command updates Babs Jensen’s description to
Post recovery
:$ /path/to/opendj/bin/ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDn uid=bjensen,ou=People,dc=example,dc=com \ --bindPassword hifalutin <<EOF dn: uid=bjensen,ou=People,dc=example,dc=com changetype: modify replace: description description: Post recovery EOF # MODIFY operation successful for DN uid=bjensen,ou=People,dc=example,dc=com
-
Read the modified entry on another replica:
$ /path/to/replica/bin/ldapsearch \ --hostname localhost \ --port 11636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=bjensen,ou=People,dc=example,dc=com \ --bindPassword hifalutin \ --baseDn dc=example,dc=com \ "(cn=Babs Jensen)" \ description dn: uid=bjensen,ou=People,dc=example,dc=com description: Post recovery
You have shown the recovery procedure succeeded.
Before deployment
When planning to deploy disaster recovery procedures, take these topics into account.
Recover before the purge delay
When recovering from backup, you must complete the recovery procedure while the backup is newer than the replication delay.
If this is not possible for all servers, recreate the remaining servers from scratch after recovering as many servers as possible and taking a new backup.
New backup after recovery
Disaster recovery resets the replication generation ID to a different format than you get when importing new directory data.
After disaster recovery, you can no longer use existing backup files for the recovered base DN. Directory servers can only replicate data under a base DN with directory servers having the same generation ID. The old backups no longer have the right generation IDs.
Instead, immediately after recovery, back up data from the recovered base DN and use the new backups going forward.
You can purge older backup files to prevent someone accidentally restoring from a backup with an outdated generation ID.
Change notifications reset
Disaster recovery clears the changelog for the recovered base DN.
If you use change number indexing for the recovered base DN, disaster recovery resets the change number.
Accounts
Account lockout
Account lockout settings are part of password policy. The server locks an account after the specified number of consecutive authentication failures. For example, users are allowed three consecutive failures before being locked out for five minutes. Failures themselves expire after five minutes.
The aim of account lockout is not to punish users who mistype their passwords. It protects the directory when an attacker attempts to guess a user password with repeated attempts to bind.
Account lockout is not transactional across a replication topology. Under normal circumstances, replication propagates lockout quickly. If replication is ever delayed, an attacker with direct access to multiple replicas could try to authenticate up to the specified number of times on each replica before being locked out on all replicas. |
The following command adds a replicated password policy to activate lockout:
$ ldapmodify \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword password << EOF
dn: cn=Lock after three failures,dc=example,dc=com
objectClass: top
objectClass: subentry
objectClass: ds-pwp-password-policy
cn: Lock after three failures
ds-pwp-password-attribute: userPassword
ds-pwp-default-password-storage-scheme: PBKDF2-HMAC-SHA256
ds-pwp-lockout-failure-expiration-interval: 5 m
ds-pwp-lockout-duration: 5 m
ds-pwp-lockout-failure-count: 3
subtreeSpecification: { base "ou=people" }
EOF
Users with this policy are locked out after three failed attempts in succession:
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
--bindPassword hifalutin \
--baseDN dc=example,dc=com \
uid=bjensen \
mail
dn: uid=bjensen,ou=People,dc=example,dc=com
mail: bjensen@example.com
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
--bindPassword fatfngrs \
--baseDN dc=example,dc=com \
uid=bjensen \
mail
The LDAP bind request failed: 49 (Invalid Credentials)
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
--bindPassword fatfngrs \
--baseDN dc=example,dc=com \
uid=bjensen \
mail
The LDAP bind request failed: 49 (Invalid Credentials)
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
--bindPassword fatfngrs \
--baseDN dc=example,dc=com \
uid=bjensen \
mail
The LDAP bind request failed: 49 (Invalid Credentials)
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
--bindPassword hifalutin \
--baseDN dc=example,dc=com \
uid=bjensen \
mail
The LDAP bind request failed: 49 (Invalid Credentials)
Account management
Disable an account
-
Make sure the user running the
manage-account
command has access to perform the appropriate operations.Kirsten Vaughan is a member of the Directory Administrators group. For this example, she must have the
password-reset
privilege, and access to edit user attributes and operational attributes:$ ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=admin \ --bindPassword password << EOF dn: uid=kvaughan,ou=People,dc=example,dc=com changetype: modify add: ds-privilege-name ds-privilege-name: password-reset dn: ou=People,dc=example,dc=com changetype: modify add: aci aci: (target="ldap:///ou=People,dc=example,dc=com")(targetattr ="*||+") (version 3.0;acl "Admins can run amok"; allow(all) groupdn = "ldap:///cn=Directory Administrators,ou=Groups,dc=example,dc=com";) EOF
Notice here that the directory superuser,
uid=admin
, assigns privileges. Any administrator with theprivilege-change
privilege can assign privileges. However, if the administrator can update administrator privileges, they can assign themselves thebypass-acl
privilege. Then they are no longer bound by access control instructions, including both user data ACIs and global ACIs. For this reason, do not assign theprivilege-change
privilege to normal administrator users. -
Set the account status to disabled:
$ manage-account \ set-account-is-disabled \ --hostname localhost \ --port 4444 \ --bindDN uid=kvaughan,ou=people,dc=example,dc=com \ --bindPassword bribery \ --operationValue true \ --targetDN uid=bjensen,ou=people,dc=example,dc=com \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin Account Is Disabled: true
Activate a disabled account
-
Clear the disabled status:
$ manage-account \ set-account-is-disabled \ --hostname localhost \ --port 4444 \ --bindDN uid=kvaughan,ou=people,dc=example,dc=com \ --bindPassword bribery \ --operationValue false \ --targetDN uid=bjensen,ou=people,dc=example,dc=com \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin Account Is Disabled: false
Account status notifications
DS servers can send mail about account status changes.
The DS server needs an SMTP server to send messages, and needs templates for the mail it sends.
By default, message templates are in English, and found in the /path/to/opendj/config/messages/
directory.
DS servers generate notifications only when the server writes to an entry
or evaluates a user entry for authentication.
A server generates account enabled and account disabled notifications
when the user account is enabled or disabled with the manage-account
command.
A server generates password expiration notifications when a user tries to bind.
For example, if you configure a notification for password expiration, that notification gets triggered when the user authenticates during the password expiration warning interval. The server does not automatically scan entries to send password expiry notifications.
DS servers implement controls that you can pass in an LDAP search to determine whether a user’s password is about to expire. Refer to Supported LDAP controls for a list. Your script or client application can send notifications based on the results of the search.
Send account status mail
-
Configure an SMTP server to use when sending messages:
$ dsconfig \ create-mail-server \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --server-name "SMTP server" \ --set enabled:true \ --set auth-username:mail.user \ --set auth-password:password \ --set smtp-server:smtp.example.com:587 \ --set trust-manager-provider:"JVM Trust Manager" \ --set use-start-tls:true \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
-
Prepare the DS server to mail users about account status.
The following example configures the server to send text-format mail messages:
$ dsconfig \ set-account-status-notification-handler-prop \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --handler-name "SMTP Handler" \ --set enabled:true \ --set email-address-attribute-type:mail \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
Notice that the server finds the user’s mail address on the attribute on the user’s entry, specified by
email-address-attribute-type
. You can also configure themessage-subject
andmessage-template-file
properties. Use interactive mode to make the changes.You find templates for messages by default under the
config/messages
directory. Edit the templates as necessary.If you edit the templates to send HTML rather than text messages, then set the advanced property,
send-email-as-html
:$ dsconfig \ set-account-status-notification-handler-prop \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --handler-name "SMTP Handler" \ --set enabled:true \ --set send-email-as-html:true \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
-
Adjust applicable password policies to use the account status notification handler you configured:
$ dsconfig \ set-password-policy-prop \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --policy-name "Default Password Policy" \ --set account-status-notification-handler:"SMTP Handler" \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
When configuring a subentry password policy, set the
ds-pwp-account-status-notification-handler
attribute, an attribute of theds-pwp-password-policy
object class.
Message templates
When editing the config/messages
templates, use the following tokens, which the server replaces with text:
%%notification-type%%
-
The name of the notification type.
%%notification-message%%
-
The message for the notification.
%%notification-user-dn%%
-
The string representation of the user DN that is the target of the notification.
%%notification-user-attr:attrname%%
-
The value of the attribute specified by attrname from the user’s entry.
If the specified attribute has multiple values, then this is the first value encountered. If the specified attribute does not have any values, then this is an empty string.
%%notification-property:propname%%
-
The value of the specified property.
If the specified property has multiple values, then this is the first value encountered. If the specified property does not have any values, then this is an empty string.
Valid propname values include the following:
-
account-unlock-time
-
new-password
-
old-password
-
password-expiration-time
-
password-policy-dn
-
seconds-until-expiration
-
seconds-until-unlock
-
time-until-expiration
-
time-until-unlock
-
Resource limits
Search limits
You can set limits on search operations:
-
The size limit sets the maximum number of entries returned for a search.
The default size limit of 1000 is set by the global server property
size-limit
.You can override the limit per user with the operational attribute,
ds-rlim-size-limit
.Search requests can include a size limit setting. The
ldapsearch
command has a--sizeLimit
option. -
The time limit defines the maximum processing time for a search operation.
The default time limit of 1 minute is set by the global server property
time-limit
.You can override the limit on a per user basis with the operational attribute,
ds-rlim-time-limit
. Times fords-rlim-time-limit
are expressed in seconds.In addition, search requests themselves can include a time limit setting. The
ldapsearch
command has an--timeLimit
option. -
The idle time limit defines how long an idle connection remains open.
No default idle time limit is set. You can set an idle time limit by using the global server property
idle-time-limit
.You can override the limit on a per user basis with the operational attribute,
ds-rlim-idle-time-limit
. Times fords-rlim-idle-time-limit
are expressed in seconds. -
The maximum number of persistent searches is set by the global server property
max-psearches
.
Set limits for a user
-
Give an administrator access to update the operational attributes related to search limits:
$ ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=admin \ --bindPassword password << EOF dn: ou=People,dc=example,dc=com changetype: modify add: aci aci: (targetattr = "ds-rlim-time-limit||ds-rlim-size-limit") (version 3.0;acl "Allow Kirsten Vaughan to manage search limits"; allow (all) (userdn = "ldap:///uid=kvaughan,ou=People,dc=example,dc=com");) EOF
-
Change the user entry to set the limits to override:
$ ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=kvaughan,ou=people,dc=example,dc=com \ --bindPassword bribery << EOF dn: uid=bjensen,ou=People,dc=example,dc=com changetype: modify add: ds-rlim-size-limit ds-rlim-size-limit: 10 EOF
When Babs Jensen performs an indexed search returning more than 10 entries, she reads the following message:
$ ldapsearch \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=bjensen,ou=people,dc=example,dc=com \ --bindPassword hifalutin \ --baseDN dc=example,dc=com \ "(sn=jensen)" # The LDAP search request failed: 4 (Size Limit Exceeded) # Additional Information: This search operation has sent the maximum of 10 entries to the client
Set limits for users in a group
-
Give an administrator the privilege to write subentries:
$ ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=admin \ --bindPassword password << EOF dn: uid=kvaughan,ou=People,dc=example,dc=com changetype: modify add: ds-privilege-name ds-privilege-name: subentry-write EOF
Notice here that the directory superuser,
uid=admin
, assigns privileges. Any administrator with theprivilege-change
privilege can assign privileges. However, if the administrator can update administrator privileges, they can assign themselves thebypass-acl
privilege. Then they are no longer bound by access control instructions, including both user data ACIs and global ACIs. For this reason, do not assign theprivilege-change
privilege to normal administrator users. -
Create an LDAP subentry to specify the limits using collective attributes:
$ ldapmodify \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=kvaughan,ou=people,dc=example,dc=com \ --bindPassword bribery << EOF dn: cn=Remove Administrator Search Limits,dc=example,dc=com objectClass: collectiveAttributeSubentry objectClass: extensibleObject objectClass: subentry objectClass: top cn: Remove Administrator Search Limits ds-rlim-size-limit;collective: 0 ds-rlim-time-limit;collective: 0 subtreeSpecification: {base "ou=people", specificationFilter "(isMemberOf=cn=Directory Administrators,ou=Groups,dc=example,dc=com)" } EOF
The
base
entry identifies the branch that holds administrator entries. For details on how subentries apply, refer to About subentry scope. -
Check the results:
$ ldapsearch \ --hostname localhost \ --port 1636 \ --useSsl \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --bindDN uid=kvaughan,ou=people,dc=example,dc=com \ --bindPassword bribery \ --baseDN uid=kvaughan,ou=people,dc=example,dc=com \ --searchScope base \ "(&)" \ ds-rlim-time-limit ds-rlim-size-limit dn: uid=kvaughan,ou=People,dc=example,dc=com ds-rlim-size-limit: 0 ds-rlim-time-limit: 0
Limit persistent searches
An LDAP persistent search maintains an open a connection that may be idle for long periods of time. Whenever a modification changes data in the search scope, the server returns a search result. The more concurrent persistent searches, the more work the server has to do for each modification:
-
Set the global property
max-psearches
to limit total concurrent persistent searches.The following example limits the maximum number of persistent searchees to 30:
$ dsconfig \ set-global-configuration-prop \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --set max-psearches:30 \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
Connection limits
Limit total connections
Each connection uses memory. On Linux systems, each connection uses an available file descriptor.
To limit the total number of concurrent client connections that the server accepts, use the global setting
max-allowed-client-connections
.
The following example sets the limit to 64K. 64K is the minimum number of file descriptors
that should be available to the DS server:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set max-allowed-client-connections:65536 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Restrict who can connect
To restrict which clients can connect to the server, use the global setting
allowed-client
,
or denied-client
.
The following example restricts access to clients from the example.com
domain:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set allowed-client:example.com \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Set these properties per Connection Handler
.
The settings on a connection handler override the global settings.
Limit connections per client
To limit the number of concurrent connections from a client, use the global settings
restricted-client
,
and restricted-client-connection-limit
.
The following example sets the limit for all clients on the 10.0.0.*
network to 1000 concurrent connections:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set restricted-client:"10.0.0.*" \
--set restricted-client-connection-limit:1000 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Set these properties per Connection Handler
.
The settings on a connection handler override the global settings.
The server applies the properties in this order:
-
If the
denied-client
property is set, the server denies connections from any client matching the settings. -
If the
restricted-client
property is set, the server checks the number of connections from any client matching the settings.If a matching client exceeds
restricted-client-connection-limit
connections, the server refuses additional connections. -
If the
allowed-client
property is set, the server allows connections from any client matching the settings. -
If none of the properties are set, the server allows connections from any client.
Idle time limits
If client applications leave connections idle for long periods, you can drop their connections by setting the global configuration property idle-time-limit. By default, no idle time limit is set.
If your network is configured to drop connections that have been idle for some time, set the DS idle time limit to a lower value than the idle time limit for the network. This helps to ensure that idle connections are shut down in orderly fashion. Setting the DS limit lower than the network limit is particularly useful with networks that drop idle connections without cleanly closing the connection and notifying the client and server.
DS servers do not enforce idle timeout for persistent searches. |
The following example sets the idle-time-limit to 24 hours:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set idle-time-limit:24h \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Request size limits
The default maximum request size is 5 MB. This is sufficient for most deployments. In cases where clients add groups with large numbers of members, requests can exceed the 5 MB limit.
The following example increases the limit to 20 MB for the LDAP connection handler:
$ dsconfig \
set-connection-handler-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--handler-name LDAP \
--set max-request-size:20mb \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
This setting affects only the size of requests, not responses.
Move a server
The following procedure moves a server to the new host new-server.example.com
.
The steps skip creation of system accounts, startup scripts, and registration as a Windows service:
-
Stop the server:
$ stop-ds
-
Renew the server certificate to account for the new hostname.
Skip this step if the server certificate is a wildcard certificate that is already valid for the new hostname.
The following command renews the server certificate generated with a deployment ID and password:
$ dskeymgr \ create-tls-key-pair \ --deploymentId $DEPLOYMENT_ID \ --deploymentIdPassword password \ --keyStoreFile /path/to/opendj/config/keystore \ --keyStorePassword:file /path/to/opendj/config/keystore.pin \ --hostname localhost \ --hostname new-server.example.com \ --subjectDn CN=DS,O=ForgeRock
For more command options, refer to dskeymgr. The default validity for the certificate is one year.
-
Find and replace the old hostname with the new hostname in the server’s configuration file,
config/config.ldif
.The following list includes configuration settings that may specify the server hostname:
-
ds-cfg-advertised-listen-address
-
ds-cfg-bootstrap-replication-server
-
ds-cfg-listen-address
-
ds-cfg-server-fqdn
-
ds-cfg-source-address
-
-
Move all files in the
/path/to/opendj
directory to the new server. -
Start the server:
$ start-ds
-
If the server you moved is referenced by others as a replication bootstrap server, update the replication bootstrap server configuration on those servers.
Performance tuning
Performance requirements
Your key performance requirement is to satisfy your users or customers with the resources available to you. Before you can solve potential performance problems, define what those users or customers expect. Determine which resources you will have to satisfy their expectations.
Service level objectives
A service level objective (SLO) is a target for a directory service level that you can measure quantitatively. If possible, base SLOs on what your key users expect from the service in terms of performance.
Define SLOs for at least the following areas:
-
Directory service response times
Directory service response times range from less than a millisecond on average, across a low latency connection on the same network, to however long it takes your network to deliver the response.
More important than average or best response times is the response time distribution, because applications set timeouts based on worst case scenarios.
An example response time performance requirement is, Directory response times must average less than 10 milliseconds for all operations except searches returning more than 10 entries, with 99.9% of response times under 40 milliseconds.
-
Directory service throughput
Directories can serve many thousands of operations per second. In fact there is no upper limit for read operations such as searches, because only write operations must be replicated. To increase read throughput, simply add additional replicas.
More important than average throughput is peak throughput. You might have peak write throughput in the middle of the night when batch jobs update entries in bulk, and peak binds for a special event or first thing Monday morning.
An example throughput performance requirement is, The directory service must sustain a mix of 5,000 operations per second made up of 70% reads, 25% modifies, 3% adds, and 2% deletes.
Ideally, you mimic the behavior of key operations during performance testing, so that you understand the patterns of operations in the throughput you need to provide.
-
Directory service availability
DS software is designed to let you build directory services that are basically available, including during maintenance and even upgrade of individual servers.
To reach very high levels of availability, you must also ensure that your operations execute in a way that preserves availability.
Availability requirements can be as lax as a best effort, or as stringent as 99.999% or more uptime.
Replication is the DS feature that allows you to build a highly available directory service.
-
Directory service administrative support
Be sure to understand how you support your users when they run into trouble.
While directory services can help you turn password management into a self-service visit to a web site, some users still need to know what they can expect if they need your help.
Creating an SLO, even if your first version consists of guesses, helps you reduce performance tuning from an open-ended project to a clear set of measurable goals for a manageable project with a definite outcome.
Resource constraints
With your SLOs in hand, inventory the server, networks, storage, people, and other resources at your disposal. Now is the time to estimate whether it is possible to meet the requirements at all.
If, for example, you are expected to serve more throughput than the network can transfer, maintain high-availability with only one physical machine, store 100 GB of backups on a 50 GB partition, or provide 24/7 support all alone, no amount of tuning will fix the problem.
When checking that the resources you have at least theoretically suffice to meet your requirements, do not forget that high availability in particular requires at least two of everything to avoid single points of failure. Be sure to list the resources you expect to have, when and how long you expect to have them, and why you need them. Make note of what is missing and why.
Server hardware
DS servers are pure Java applications, making them very portable. DS servers tend to perform best on single-board, x86 systems due to low memory latency.
Storage
High-performance storage is essential for handling high-write throughput. When the database stays fully cached in memory, directory read operations do not result in disk I/O. Only writes result in disk I/O. You can further improve write performance by using solid-state disks for storage or file system cache.
DS directory servers are designed to work with local storage for database backends. Do not use network file systems, such as NFS, where there is no guarantee that a single process has access to files. Storage area networks (SANs) and attached storage are fine for use with DS directory servers. |
Regarding database size on disk, sustained write traffic can cause the database to grow to more than twice its initial size on disk. This is normal behavior. The size on disk does not impact the DB cache size requirements.
Linux file systems
Write barriers and journaling mode for Linux file systems help avoid directory database file corruption. They make sure writes to the file system are ordered even after a crash or power failure. Make sure these features are enabled.
Some Linux distributions permanently enable write barriers. There is no administrative action to take.
Other Linux systems leave the decision to you.
If your Linux system lets you configure write barriers and journaling mode for the file system,
refer to the options for your file system in the mount
command manual page for details on enabling them.
Performance tests
Even if you do not need high availability, you still need two of everything, because your test environment needs to mimic your production environment as closely as possible.
In your test environment, set up DS servers just as you do in production. Conduct experiments to determine how to best meet your SLOs.
The following command-line tools help with basic performance testing:
-
The makeldif command generates sample data with great flexibility.
-
The
addrate
command measures add and delete throughput and response time. -
The
authrate
command measures bind throughput and response time. -
The
modrate
command measures modification throughput and response time. -
The
searchrate
command measures search throughput and response time.
All *rate
commands display response time distributions measurements,
and support testing at specified levels of throughput.
For additional precision when evaluating response times, use the global configuration setting etime-resolution. To change elapsed processing time resolution from milliseconds (default) to nanoseconds:
$ dsconfig \
set-global-configuration-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--set etime-resolution:nanoseconds \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
The etime
, recorded in the server access log, indicates the elapsed time to process the request.
The etime
starts when the decoded operation is available to be processed by a worker thread.
Test performance with your production-ready configuration. If, however, you simply want to demonstrate top performance, take the following points into account:
-
Incorrect JVM tuning slows down server and tool performance.
Make sure the JVM is tuned for best performance. For details, refer to Java settings.
-
Unfiltered access logs record messages for each client request. Turn off full access logging.
For example, set
enabled:false
for theJson File-Based Access Logger
log publisher, and any other unfiltered log publishers that are enabled. -
Secure connections are recommended, and they can be costly.
Set
require-secure-authentication:false
in the password policies governing the bind entries, and bind using insecure connections.
Performance settings
Use the following suggestions when your tests show that DS performance is lacking, even though you have the right underlying network, hardware, storage, and system resources in place.
Maximum open files
DS servers must open many file descriptors when handling thousands of client connections.
Linux systems often set a limit of 1024 per user. That setting is too low to accept thousands of client connections.
Make sure the server can use at least 64K (65536) file descriptors.
For example, when running the server as user opendj
on a Linux system
that uses /etc/security/limits.conf
to set user level limits,
set soft and hard limits by adding these lines to the file:
opendj soft nofile 65536
opendj hard nofile 131072
The example above assumes the system has enough file descriptors available overall. Check the Linux system overall maximum as follows:
$ cat /proc/sys/fs/file-max
204252
Linux page caching
Default Linux virtual memory settings cause significant buildup of dirty data pages before flushing them. When the kernel finally flushes the pages to disk, the operation can exhaust the disk I/O for up to several seconds. Application operations waiting on the file system to synchronize to disk are blocked.
The default virtual memory settings can therefore cause DS server operations to block for seconds at a time. Symptoms included high outlier etimes, even for very low average etimes. For sustained high loads, such as import operations, the server has to maintain thousands of open file descriptors.
To avoid these problems, tune Linux page caching.
As a starting point for testing and tuning, set vm.dirty_background_bytes
to one quarter of the disk I/O per second,
and vm.dirty_expire_centisecs
to 1000 (10 seconds) using the sysctl
command.
This causes the kernel to flush more often, and limits the pauses to a maximum of 250 milliseconds.
For example, if the disk I/O is 80 MB/second for writes, the following example shows an appropriate starting point.
It updates the /etc/sysctl.conf
file to change the setting permanently,
and uses the sysctl -p
command to reload the settings:
$ echo vm.dirty_background_bytes=20971520 | sudo tee -a /etc/sysctl.conf
[sudo] password for admin:
$ echo vm.dirty_expire_centisecs=1000 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
vm.dirty_background_bytes = 20971520
vm.dirty_expire_centisecs = 1000
Be sure to test and adjust the settings for your deployment.
For additional details, refer to the Oracle documentation on
Linux Page Cache Tuning,
and the Linux sysctl
command
virtual memory kernel reference.
Java settings
Default Java settings let you evaluate DS servers using limited system resources. For production systems, test and run with a tuned JVM.
To apply JVM settings, either:
|
-
Use the most recent supported Java environment.
Refer to the release notes section on Java for details.
-
Set the maximum heap size with
-Xmx<size>
.Use at least a 2 GB heap unless your data set is small.
For additional details, refer to Cache internal nodes.
For Java 17, there is no need to set the minimum heap size. If you use Java 11, set the minimum heap size to the same value as the maximum heap size.
-
Use the default garbage collection (GC) settings, equivalent to
-XX:+UseG1GC
. -
Set the maximum tenuring threshold to reduce unnecessary copying with
-XX:MaxTenuringThreshold=1
.This option sets the maximum number of GC cycles an object stays in survivor spaces before it is promoted into the old generation space. The recommended setting reduces the new generation GC frequency and duration. The JVM quickly promotes long-lived objects to the old generation space, rather than letting them accumulate in new generation survivor spaces, copying them for each GC cycle.
-
(Optional) Review the following additional details for specific use cases:
-XX:+DisableExplicitGC
-
When using JMX, add this option to the list of
start-ds.java-args
arguments to avoid periodic full GC events.JMX is based on RMI, which uses references to objects. By default, the JMX client and server perform a full GC periodically to clean up stale references. As a result, the default settings cause JMX to cause a full GC every hour.
Avoid using this argument with
import-ldif.offline.java-args
or when using theimport-ldif
command. The import process uses GC to manage memory and references to memory-mapped files. -Xlog:gc=level:file
-
When diagnosing JVM tuning problems, log GC messages. You can turn the option off when everything is running smoothly.
Always specify the output file for the GC log. Otherwise, the JVM logs the messages to the
opendj/logs/server.out
file, mixing them with other messages, such as stack traces from thesupportextract
command.For example,
-Xlog:gc=info:file=/path/to/gc.log
logs informational GC messages to the file,/path/to/gc.log
.For details, use the
java -Xlog:help
command. -XX:TieredStopAtLevel=1
-
When trying to reduce startup time for short-lived client tools, such as the
ldapsearch
command, use this setting as shown.
ForgeRock does not recommend using ZGC or huge pages at this time.
Data storage settings
By default, DS servers compress attribute descriptions and object class sets to reduce data size. This is called compact encoding.
By default, DS servers do not compress entries stored in its backend database.
If your entries hold values that compress well, such as text, you can gain space.
Set the backend property entries-compressed:true
, and reimport the data from LDIF.
The DS server compresses entries before writing them to the database:
$ dsconfig \
set-backend-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--backend-name dsEvaluation \
--set entries-compressed:true \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
$ import-ldif \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--ldifFile backup.ldif \
--backendID dsEvaluation \
--includeBranch dc=example,dc=com \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin
DS directory servers do not proactively rewrite all entries after you change the settings. To force the DS server to compress all entries, you must import the data from LDIF.
LDIF import settings
By default, the temporary directory used for scratch files is opendj/import-tmp
.
Use the import-ldif --tmpDirectory
option to set this directory to a tmpfs
file system,
such as /tmp
.
If you are certain your LDIF contains only valid entries with correct syntax, you can skip schema validation.
Use the import-ldif --skipSchemaValidation
option.
Database cache settings
By default, DS directory servers:
|
If you require fine-grained control over JE backend cache settings, you can configure the amount of memory requested for database cache per database backend:
-
Configure
db-cache-percent
ordb-cache-size
for each JE backend.db-cache-percent
-
Percentage of JVM memory to allocate to the database cache for the backend.
If the directory server has multiple database backends, the total percent of JVM heap used must remain less than 100 (percent), and must leave space for other uses.
Default: 50 (percent)
db-cache-size
-
JVM memory to allocate to the database cache.
This is an alternative to
db-cache-percent
. If you set its value larger than 0, then it takes precedence overdb-cache-percent
.Default: 0 MB
-
Set the global property
je-backend-shared-cache-enabled:false
. -
Restart the server for the changes to take effect.
Cache internal nodes
A JE backend has a B-tree data structure. A B-tree consists of nodes that can have children. Nodes with children are internal nodes. Nodes without children are leaf nodes.
The directory stores data in key-value pairs. Internal nodes hold the keys and can hold small values. Leaf nodes hold the values. One internal node usually holds keys to values in many leaf nodes. A B-tree has many more leaf nodes than internal nodes.
To read a value by its key, the backend traverses all internal nodes on the branch from the B-tree root to the leaf node holding the value. The closer a node is to the B-tree root, the more likely the backend must access it to get to the value. In other words, the backend accesses internal nodes more often than leaf nodes.
When a backend accesses a node, it loads the node into the DB cache. Loading a node because it wasn’t in cache is a cache miss. When you first start DS, all requests result in cache misses until the server loads active nodes.
As the DB cache fills, the backend makes space to load nodes by evicting nodes from the cache. The backend evicts leaf nodes, then least recently used internal nodes. As a last resort, the backend evicts even recently used internal nodes with changes not yet synced to storage.
The next time the backend accesses an evicted node, it must load the node from storage. Storage may mean the file system cache, or it may mean a disk. Reading from memory can be orders of magnitude faster than reading from disk. For the best DB performance, cache the nodes the DB accesses most often, which are the internal nodes.
Once DS has run for some time and active nodes are in cache, watch the cache misses for internal nodes. DS has "warmed up" and the active nodes are in the cache. The number of evicted internal nodes should remain constant. When the cache size is right, and no sudden changes occur in access patterns, the number of cache misses for internal nodes should stop growing:
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword password \
--baseDN cn=monitor \
"(objectClass=ds-monitor-backend-db)" \
ds-mon-db-cache-evict-internal-nodes-count \
ds-mon-db-cache-misses-internal-nodes
dn: ds-cfg-backend-id=dsEvaluation,cn=backends,cn=monitor
ds-mon-db-cache-evict-internal-nodes-count: <number>
ds-mon-db-cache-misses-internal-nodes: <number>
If ds-mon-db-cache-evict-internal-nodes-count
is greater than 0
and growing,
or ds-mon-db-cache-misses-internal-nodes
continues to grow even after DS has warmed up,
DS is evicting internal nodes from the DB cache.
If you can rule out big changes in access cache patterns like large unindexed searches,
DS does not have enough space for the DB cache.
Increase the DB cache size and add more RAM to your system if necessary.
If adding RAM isn’t an option, increase the maximum heap size (-Xmx
) to optimize RAM allocation.
Estimate minimum DB cache size
This section explains how to estimate the minimum DB cache size.
The examples below use a directory server with a 10 million entry
dsEvaluation
backend.
The backend holds entries generated using the --set ds-evaluation/generatedUsers:10,000,000
setup option.
Before estimating DB cache size for your deployment:
|
After simulating realistic traffic for some time, stop the server.
Use the output from the backendstat
command to estimate the required DB cache size
JE DbCacheSize
tool
together to estimate the required DB cache size:
# Stop the server before using backendstat:
$ stop-ds
$ backendstat list-raw-dbs --backendId dsEvaluation
Raw DB Name Total Keys Keys Size Values Size Total Size
-----------------------------------------------------------------------------------------------------------------------------------------------
/compressed_schema/compressed_attributes 54 54 837 891
/compressed_schema/compressed_object_classes 18 18 938 956
/dc=com,dc=example/aci.presence 1 1 3 4
/dc=com,dc=example/cn.caseIgnoreMatch 10000165 139242470 47887210 187129680
/dc=com,dc=example/cn.caseIgnoreSubstringsMatch:6 858657 5106079 204936276 210042355
/dc=com,dc=example/counter.objectClass.big.objectIdentifierMatch 2 34 2 36
/dc=com,dc=example/counter.userPassword.big.passwordStorageSchemeEqualityMatch 0 0 0 0
/dc=com,dc=example/dn2id 10000181 268892913 80001448 348894361
/dc=com,dc=example/ds-certificate-fingerprint.caseIgnoreMatch 0 0 0 0
/dc=com,dc=example/ds-certificate-subject-dn.distinguishedNameMatch 1 18 3 21
/dc=com,dc=example/ds-sync-conflict.distinguishedNameMatch 0 0 0 0
/dc=com,dc=example/ds-sync-hist.changeSequenceNumberOrderingMatch 0 0 0 0
/dc=com,dc=example/entryUUID.uuidMatch 9988518 39954072 47871653 87825725
/dc=com,dc=example/givenName.caseIgnoreMatch 8614 51690 20017387 20069077
/dc=com,dc=example/givenName.caseIgnoreSubstringsMatch:6 19651 97664 48312525 48410189
/dc=com,dc=example/id2childrencount 8 26 14 40
/dc=com,dc=example/id2entry 10000181 80001448 5379599451 5459600899
/dc=com,dc=example/json.caseIgnoreJsonQueryMatch 4 56 8 64
/dc=com,dc=example/jsonToken.extensibleJsonEqualityMatch:caseIgnoreStrings:ignoreWhiteSpace:/id 2 34 4 38
/dc=com,dc=example/mail.caseIgnoreMatch 10000152 238891751 47887168 286778919
/dc=com,dc=example/mail.caseIgnoreSubstringsMatch:6 1222798 7336758 112365097 119701855
/dc=com,dc=example/member.distinguishedNameMatch 1 40 2 42
/dc=com,dc=example/oauth2Token.caseIgnoreOAuth2TokenQueryMatch 4 74 10 84
/dc=com,dc=example/objectClass.big.objectIdentifierMatch 6 156 0 156
/dc=com,dc=example/objectClass.objectIdentifierMatch 24 396 395 791
/dc=com,dc=example/referral 0 0 0 0
/dc=com,dc=example/sn.caseIgnoreMatch 13457 92943 20027045 20119988
/dc=com,dc=example/sn.caseIgnoreSubstringsMatch:6 41585 219522 73713958 73933480
/dc=com,dc=example/state 26 1335 25 1360
/dc=com,dc=example/telephoneNumber.telephoneNumberMatch 9989952 109889472 47873522 157762994
/dc=com,dc=example/telephoneNumber.telephoneNumberSubstringsMatch:6 1111110 6543210 221281590 227824800
/dc=com,dc=example/uid.caseIgnoreMatch 10000152 118889928 47887168 166777096
/dc=com,dc=example/uniqueMember.uniqueMemberMatch 10 406 21 427
/dc=com,dc=example/userPassword.big.passwordStorageSchemeEqualityMatch 0 0 0 0
Total: 34
# Calculate the sum of total keys, the average key size, and the average value size.
# Sum of total keys: 73255334
# Average key size: sum of key sizes/sum of total keys = 1015212568 / 73255334 ~= 13.86
# Average value size: sum of values sizes/sum of total keys = 6399663760 / 73255334 ~= 87.36
# Use the results rounded to the nearest integer as arguments to the DbCacheSize tool:
$ java -cp /path/to/opendj/lib/opendj.jar com.sleepycat.je.util.DbCacheSize \
-records 73255334 -key 14 -data 87
=== Environment Cache Overhead ===
3,158,773 minimum bytes
To account for JE daemon operation, record locks, HA network connections, etc,
a larger amount is needed in practice.
=== Database Cache Size ===
Number of Bytes Description
--------------- -----------
2,953,929,424 Internal nodes only
12,778,379,408 Internal nodes and leaf nodes
For further information see the DbCacheSize javadoc.
The resulting recommendation for caching Internal nodes only
is 2,953,929,424 bytes (~ 3 GB) in this example.
This setting for DB cache includes space for all internal nodes, including those with keys and data.
To cache all DB data, Internal nodes and leaf nodes
, would require 12,778,379,408 (~13 GB).
Round up when configuring backend settings for db-cache-percent
or db-cache-size
.
If the system in this example has 8 GB available memory, use the default setting of db-cache-percent: 50
.
(50% of 8 GB is 4 GB, which is larger than the minimum estimate.)
Database log file settings
With default settings, if the database has more than 200 files on disk, then the JE backend must start closing one log file in order to open another. This has serious impact on performance when the file cache starts to thrash.
Having the JE backend open and close log files from time to time is okay. Changing the settings is only necessary if the JE backend has to open and close the files very frequently.
A JE backend stores data on disk in append-only log files. The maximum size of each log file is configurable. A JE backend keeps a configurable maximum number of log files open, caching file handles to the log files. The relevant JE backend settings are the following:
db-log-file-max
-
Maximum size of a database log file.
Default: 1 GB
db-log-filecache-size
-
File handle cache size for database log files.
Default: 200
With these defaults, if the size of the database reaches 200 GB on disk (1 GB x 200 files),
the JE backend must close one log file to open another.
To avoid this situation, increase db-log-filecache-size
until the JE backend can cache file handles to all its log files.
When changing the settings, make sure the maximum number of open files is sufficient.
Log settings
Debug-level log settings trace the internal workings of DS servers, and should be used sparingly. Be careful when activating debug-level logging in high-performance deployments.
In general, leave other logs active for production environments to help troubleshoot any issues that arise.
For servers handling 100,000 operations per second or more, the access log can be a performance bottleneck. Each client request results in at least one access log message. Test whether disabling the access log improves performance in such cases.
The following command disables the JSON-based LDAP access logger:
$ dsconfig \
set-log-publisher-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--publisher-name "Json File-Based Access Logger" \
--set enabled:false \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
The following command disables the HTTP access logger:
$ dsconfig \
set-log-publisher-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--publisher-name "File-Based HTTP Access Logger" \
--set enabled:false \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Changelog settings
To let legacy applications get update notifications by change number, as described in Align draft change numbers, enable change number indexing.
Change number indexing requires more CPU, disk access, and storage. Enable it only when applications require change number-based browsing.
Troubleshooting
Define the problem
To solve your problem, save time by clearly defining it first. A problem statement compares the difference between observed behavior and expected behavior:
-
What exactly is the problem?
What is the behavior you expected?
What is the behavior you observed?
-
How do you reproduce the problem?
-
When did the problem begin?
Under similar circumstances, when does the problem not occur?
-
Is the problem permanent?
Intermittent?
Is it getting worse? Getting better? Staying the same?
Performance
Before troubleshooting performance, make sure:
-
The system meets the DS installation requirements.
-
The performance expectations are reasonable.
For example, a deployment can use password policies with cost-based, resource-intensive password storage schemes such as Argon2, Bcrypt, or PBKDF2. This protects passwords at the cost of slow LDAP simple binds or HTTP username/password authentications and lower throughput.
When directory operations take too long, meaning request latency is high, fix the problem first in your test or staging environment. Perform these steps in order and stop when you find a fix:
-
Check for unindexed searches and prevent them when possible.
Unindexed searches are expensive operations, particularly for large directories. When unindexed searches consume the server’s resources, performance suffers for concurrent operations and for later operations if an unindexed search causes widespread changes to database and file system caches.
-
Check performance settings for the server including JVM heap size and DB cache size.
Try adding more RAM if memory seems low.
-
Read the request queue monitoring statistics over LDAP or over HTTP.
If many requests are in the queue, the troubleshooting steps are different for read and write operations. Read and review the request statistics available over LDAP or over HTTP.
If you persistently have many:
-
Pending read requests, such as unindexed searches or big searches, try adding CPUs.
-
Pending write requests, try adding IOPS, such as faster or higher throughput disks.
-
Installation problems
Use the logs
Installation and upgrade procedures result in a log file tracing the operation. The command output shows a message like the following:
See opendj-setup-profile-*.log for a detailed log of the failed operation.
Antivirus interference
Prevent antivirus and intrusion detection systems from interfering with DS software.
Before using DS software with antivirus or intrusion detection software, consider the following potential problems:
- Interference with normal file access
-
Antivirus and intrusion detection systems that perform virus scanning, sweep scanning, or deep file inspection are not compatible with DS file access, particularly write access.
Antivirus and intrusion detection software have incorrectly marked DS files as suspect to infection, because they misinterpret normal DS processing.
Prevent antivirus and intrusion detection systems from scanning DS files, except these folders:
C:\path\to\opendj\bat\
-
Windows command-line tools
/path/to/opendj/bin/
-
Linux command-line tools
/path/to/opendj/extlib/
-
Optional
.jar
files used by custom plugins /path/to/opendj/lib/
-
Scripts and libraries shipped with DS servers
- Port blocking
-
Antivirus and intrusion detection software can block ports that DS uses to provide directory services.
Make sure that your software does not block the ports that DS software uses. For details, refer to Administrative access.
- Negative performance impact
-
Antivirus software consumes system resources, reducing resources available to other services including DS servers.
Running antivirus software can therefore have a significant negative impact on DS server performance. Make sure that you test and account for the performance impact of running antivirus software before deploying DS software on the same systems.
JE initialization
When starting a directory server on a Linux system, make sure the server user can watch enough files. If the server user cannot watch enough files, you might read an error message in the server log like this:
InitializationException: The database environment could not be opened: com.sleepycat.je.EnvironmentFailureException: (JE version) /path/to/opendj/db/userData or its sub-directories to WatchService. UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects. Environment is invalid and must be closed.
File notification
A directory server backend database monitors file events.
On Linux systems, backend databases use the inotify API for this purpose.
The kernel tunable fs.inotify.max_user_watches
indicates the maximum number of files
a user can watch with the inotify API.
Make sure this tunable is set to at least 512K:
$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288
If this tunable is set lower than that, update the /etc/sysctl.conf
file to change the setting permanently,
and use the sysctl -p
command to reload the settings:
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
[sudo] password for admin:
$ sudo sysctl -p
fs.inotify.max_user_watches = 524288
NoSuchAlgorithmException
When running the dskeymgr create-deployment-id
or setup
command
on an operating system with no support for the PBKDF2WithHmacSHA256
SecretKeyFactory
algorithm,
the command displays this error:
NoSuchAlgorithmException: PBKDF2WithHmacSHA256 SecretKeyFactory not available
This can occur on operating systems where the default settings limit the available algorithms.
To fix the issue, enable support for the algorithm and run the command again.
Forgotten superuser password
By default, DS servers store the entry for the directory superuser in an LDIF backend. Edit the file to reset the password:
-
Generate the encoded version of the new password:
$ encode-password --storageScheme PBKDF2-HMAC-SHA256 --clearPassword password {PBKDF2-HMAC-SHA256}10<hash>
-
Stop the server while you edit the LDIF file for the backend:
$ stop-ds
-
Replace the existing password with the encoded version.
In the
db/rootUser/rootUser.ldif
file, carefully replace theuserPassword
value with the new, encoded password:dn: uid=admin ... uid: admin userPassword:
Trailing whitespace is significant in LDIF. Take care not to add any trailing whitespace at the end of the line.
-
Restart the server:
$ start-ds
-
Verify that you can use the directory superuser account with the new password:
$ status \ --bindDn uid=admin \ --bindPassword password \ --hostname localhost \ --port 4444 \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --script-friendly ... "isRunning" : true,
Debug-level logging
DS error log message severity levels are:
-
ERROR
(highest severity) -
WARNING
-
NOTICE
-
INFO
-
DEBUG
(lowest severity)
By default, DS error log severity levels are set as follows:
-
Log
ERROR
,WARNING
,NOTICE
, andINFO
replication (SYNC
) messages. -
Log
ERROR
,WARNING
, andNOTICE
messages for other message categories.
You can change these settings when necessary to log debug-level messages.
DS debug-level logging can generate a high volume of messages. Use debug-level logging very sparingly on production systems. |
-
Choose the category you want to debug:
Category Description BACKEND
Server backends
BACKUP
Backup procedures
CONFIG
Configuration management
CORE
Core server operations
DEFAULT
Messages with no specific category
EXTENSIONS
Reserved for custom extensions
EXTERNAL
External libraries
JVM
Java virtual machine information
LOGGING
Server log publishers
PLUGIN
Server plugins
PROTOCOL
Server protocols
PROTOCOL.ASN1
ASN.1 encoding
PROTOCOL.HTTP
HTTP
PROTOCOL.JMX
JMX
PROTOCOL.LDAP
LDAP
PROTOCOL.LDAP_CLIENT
LDAP SDK client features
PROTOCOL.LDAP_SERVER
LDAP SDK server features
PROTOCOL.LDIF
LDIF
PROTOCOL.SASL
SASL
PROTOCOL.SMTP
SMTP
PROTOCOL.SNMP
SNMP
PROTOCOL.SSL
SSL and TLS
SCHEMA
LDAP schema
SECURITY
Security features
SECURITY.AUTHENTICATION
Authentication
SECURITY.AUTHORIZATION
Access control and privileges
SERVICE_DISCOVERY
Service discovery
SYNC
Replication
SYNC.CHANGELOG
Replication changelog
SYNC.CHANGENUMBER
Replication change number and change number index
SYNC.CONNECTIONS
Replication connections
SYNC.HEARTBEAT
Replication heartbeat checks
SYNC.LIFECYCLE
Replication lifecycle
SYNC.PROTOCOL_MSGS
Replication protocol messages excluding updates and heartbeat checks
SYNC.PURGE
Replication changelog and historical data purge events
SYNC.REPLAY
Replication replays and conflicts
SYNC.STATE
Replication state changes including generation ID
SYNC.TOPOLOGY
Replication topology
SYNC.UPDATE_MSGS
Replication update messages
TASK
Server tasks
TOOLS
Command-line tools
A monitoring user can read the list of supported categories over LDAP.
-
Override the error log level specifically for the category or categories of interest.
The following example enables debug-level logging for the replication lifecycle. As debug-level logging is of lower severity than the defaults, all the default log levels remain in effect:
$ dsconfig \ set-log-publisher-prop \ --add override-severity:SYNC.LIFECYCLE=DEBUG \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --publisher-name "File-Based Error Logger" \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
The server immediately begins to write additional messages to the error log.
-
Read the messages:
$ tail -f /path/to/opendj/logs/errors
-
Restore the default settings as soon as debug-level logging is no longer required:
$ dsconfig \ set-log-publisher-prop \ --remove override-severity:SYNC.LIFECYCLE=DEBUG \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --publisher-name "File-Based Error Logger" \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
Lockdown mode
Misconfiguration can put the DS server in a state where you must prevent users and applications from accessing the directory until you have fixed the problem.
DS servers support lockdown mode .
Lockdown mode permits connections only on the loopback address,
and permits only operations requested by superusers, such as uid=admin
.
To put the DS server into lockdown mode, the server must be running. You cause the server to enter lockdown mode by starting a task. Notice that the modify operation is performed over the loopback address (accessing the DS server on the local host):
$ ldapmodify \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword password << EOF
dn: ds-task-id=Enter Lockdown Mode,cn=Scheduled Tasks,cn=tasks
objectClass: top
objectClass: ds-task
ds-task-id: Enter Lockdown Mode
ds-task-class-name: org.opends.server.tasks.EnterLockdownModeTask
EOF
The DS server logs a notice message in logs/errors
when lockdown mode takes effect:
...msg=Lockdown task Enter Lockdown Mode finished execution
Client applications that request operations get a message concerning lockdown mode:
$ ldapsearch \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--baseDN "" \
--searchScope base \
"(objectclass=*)" \
+
# The LDAP search request failed: 53 (Unwilling to Perform)
# Additional Information: Rejecting the requested operation because the server is in lockdown mode and will only accept requests from root users over loopback connections
Leave lockdown mode by starting a task:
$ ldapmodify \
--hostname localhost \
--port 1636 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword password << EOF
dn: ds-task-id=Leave Lockdown Mode,cn=Scheduled Tasks,cn=tasks
objectClass: top
objectClass: ds-task
ds-task-id: Leave Lockdown Mode
ds-task-class-name: org.opends.server.tasks.LeaveLockdownModeTask
EOF
The DS server logs a notice message when leaving lockdown mode:
...msg=Leave Lockdown task Leave Lockdown Mode finished execution
LDIF import
-
By default, DS directory servers check that entries you import match the LDAP schema.
You can temporarily bypass this check with the
import-ldif --skipSchemaValidation
option. -
By default, DS servers ensure that entries have only one structural object class.
You can relax this behavior with the advanced global configuration property,
single-structural-objectclass-behavior
.This can be useful when importing data exported from Sun Directory Server.
For example, warn when entries have more than one structural object class, rather than rejecting them:
$ dsconfig \ set-global-configuration-prop \ --hostname localhost \ --port 4444 \ --bindDN uid=admin \ --bindPassword password \ --set single-structural-objectclass-behavior:warn \ --usePkcs12TrustStore /path/to/opendj/config/keystore \ --trustStorePassword:file /path/to/opendj/config/keystore.pin \ --no-prompt
-
By default, DS servers check syntax for several attribute types. Relax this behavior using the advanced global configuration property,
invalid-attribute-syntax-behavior
. -
Use the
import-ldif -R rejectFile --countRejects
options to log rejected entries and to return the number of rejected entries as the command’s exit code.
Once you resolve the issues, reinstate the default behavior to avoid importing bad data.
Security problems
Incompatible Java versions
Due to a change in Java APIs,
the same DS deployment ID generates different CA key pairs with Java 11 and Java 17 and later.
When running the dskeymgr
and setup
commands,
use the same Java environment everywhere in the deployment.
Using different Java versions is a problem if you use deployment ID-based CA certificates.
Replication breaks, for example, when you use the setup
command
for a new server with a more recent version of Java than was used to set up existing servers.
The error log includes a message such as the following:
...category=SYNC severity=ERROR msgID=119 msg=Directory server DS(server_id) encountered an unexpected error while connecting to replication server host:port for domain "base_dn": ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
To work around the issue, follow these steps:
-
Update all DS servers to use the same Java version.
Make sure you have a required Java environment installed on the system.
If your default Java environment is not appropriate, use one of the following solutions:
-
Edit the
default.java-home
setting in theopendj/config/java.properties
file. -
Set
OPENDJ_JAVA_HOME
to the path to the correct Java environment. -
Set
OPENDJ_JAVA_BIN
to the absolute path of thejava
command.
-
-
Export CA certificates generated with the different Java versions.
-
Export the CA certificate from an old server:
$ keytool \ -exportcert \ -alias ca-cert \ -keystore /path/to/old-server/config/keystore \ -storepass:file /path/to/old-server/config/keystore.pin \ -file java11-ca-cert.pem
-
Export the CA certificate from a new server:
$ keytool \ -exportcert \ -alias ca-cert \ -keystore /path/to/new-server/config/keystore \ -storepass:file /path/to/new-server/config/keystore.pin \ -file java17-ca-cert.pem
-
-
On all existing DS servers, import the new CA certificate:
$ keytool \ -importcert \ -trustcacerts \ -alias alt-ca-cert \ -keystore /path/to/old-server/config/keystore \ -storepass:file /path/to/old-server/config/keystore.pin \ -file java17-ca-cert.pem \ -noprompt
-
On all new DS servers, import the old CA certificate:
$ keytool \ -importcert \ -trustcacerts \ -alias alt-ca-cert \ -keystore /path/to/new-server/config/keystore \ -storepass:file /path/to/new-server/config/keystore.pin \ -file java11-ca-cert.pem \ -noprompt
The servers reload their keystores dynamically and replication works as expected.
Certificate-based authentication
Replication uses TLS to protect directory data on the network. Misconfiguration can cause replicas to fail to connect due to handshake errors. This leads to repeated error log messages such as the following:
...msg=Replication server accepted a connection from address to local address address but the SSL handshake failed. This is probably benign, but may indicate a transient network outage or a misconfigured client application connecting to this replication server. The error was: Received fatal alert: certificate_unknown
You can collect debug trace messages to help determine the problem.
To display the TLS debug messages, start the server with javax.net.debug
set:
$ OPENDJ_JAVA_ARGS="-Djavax.net.debug=all" start-ds
The debug trace settings result in many, many messages. To resolve the problem, review the output of starting the server, looking in particular for handshake errors.
If the chain of trust for your PKI is broken somehow, consider renewing or replacing keys, as described in Key management. Make sure that trusted CA certificates are configured as expected.
FIPS and key wrapping
DS servers use shared asymmetric keys to protect shared symmetric secret keys for data encryption.
By default, DS uses direct encryption to protect the secret keys.
When using a FIPS-compliant security provider that doesn’t allow direct encryption, such as Bouncy Castle,
change the Crypto Manager configuration to set the advanced property, key-wrapping-mode: WRAP
.
With this setting, DS uses wrap mode to protect the secret keys in a compliant way.
Compromised keys
How you handle the problem depends on which key was compromised:
-
For keys generated by the server, or with a deployment ID and password, refer to Retire secret keys.
-
For a private key whose certificate was signed by a CA, contact the CA for help. The CA might choose to publish a certificate revocation list (CRL) that identifies the certificate of the compromised key.
Replace the key pair that has the compromised private key.
-
For a private key whose certificate was self-signed, replace the key pair that has the compromised private key.
Make sure the clients remove the compromised certificate from their truststores. They must replace the certificate of the compromised key with the new certificate.
Client problems
Use the logs
By default, DS servers record messages for LDAP client operations
in the logs/ldap-access.audit.json
log file.
Show example log messages
In the access log, each message is a JSON object. This example formats each message to make it easier to read:
{
"eventName": "DJ-LDAP",
"client": {
"ip": "<clientIp>",
"port": 12345
},
"server": {
"ip": "<serverIp>",
"port": 1636
},
"request": {
"protocol": "LDAPS",
"operation": "BIND",
"connId": 3,
"msgId": 1,
"version": "3",
"dn": "uid=kvaughan,ou=people,dc=example,dc=com",
"authType": "SIMPLE"
},
"transactionId": "<uuid>",
"response": {
"status": "SUCCESSFUL",
"statusCode": "0",
"elapsedTime": 1,
"elapsedQueueingTime": 0,
"elapsedProcessingTime": 1,
"elapsedTimeUnits": "MILLISECONDS",
"additionalItems": {
"ssf": 128
}
},
"userId": "uid=kvaughan,ou=People,dc=example,dc=com",
"timestamp": "<timestamp>",
"_id": "<uuid>"
}
{
"eventName": "DJ-LDAP",
"client": {
"ip": "<clientIp>",
"port": 12345
},
"server": {
"ip": "<serverIp>",
"port": 1636
},
"request": {
"protocol": "LDAPS",
"operation": "SEARCH",
"connId": 3,
"msgId": 2,
"dn": "dc=example,dc=com",
"scope": "sub",
"filter": "(uid=bjensen)",
"attrs": ["cn"]
},
"transactionId": "<uuid>",
"response": {
"status": "SUCCESSFUL",
"statusCode": "0",
"elapsedTime": 3,
"elapsedQueueingTime": 0,
"elapsedProcessingTime": 3,
"elapsedTimeUnits": "MILLISECONDS",
"nentries": 1,
"entrySize": 591
},
"userId": "uid=kvaughan,ou=People,dc=example,dc=com",
"timestamp": "<timestamp>",
"_id": "<uuid>"
}
{
"eventName": "DJ-LDAP",
"client": {
"ip": "<clientIp>",
"port": 12345
},
"server": {
"ip": "<serverIp>",
"port": 1636
},
"request": {
"protocol": "LDAPS",
"operation": "UNBIND",
"connId": 3,
"msgId": 3
},
"transactionId": "<uuid>",
"timestamp": "<timestamp>",
"_id": "<uuid>"
}
{
"eventName": "DJ-LDAP",
"client": {
"ip": "<clientIp>",
"port": 12345
},
"server": {
"ip": "<serverIp>",
"port": 1636
},
"request": {
"protocol": "LDAPS",
"operation": "DISCONNECT",
"connId": 3
},
"transactionId": "0",
"response": {
"status": "SUCCESSFUL",
"statusCode": "0",
"elapsedTime": 0,
"elapsedTimeUnits": "MILLISECONDS",
"reason": "Client Unbind"
},
"timestamp": "<timestamp>",
"_id": "<uuid>"
}
For details about the messages format, refer to Common ForgeRock access logs.
By default, the server does not log internal LDAP operations corresponding to HTTP requests. To match HTTP client operations to internal LDAP operations:
-
Prevent the server from suppressing log messages for internal operations.
Set
suppress-internal-operations:false
on the LDAP access log publisher. -
Match the
request/connId
field in the HTTP access log with the same field in the LDAP access log.
Client access
To help diagnose client errors due to access permissions, refer to Effective rights.
Simple paged results
For some versions of Linux, you read a message in the DS access logs such as the following:
The request control with Object Identifier (OID) "1.2.840.113556.1.4.319" cannot be used due to insufficient access rights
This message means clients are trying to use the simple paged results control without authenticating. By default, a global ACI allows only authenticated users to use the control.
To grant anonymous (unauthenticated) user access to the control, add a global ACI for anonymous use of the simple paged results control:
$ dsconfig \
set-access-control-handler-prop \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword "password" \
--add global-aci:"(targetcontrol=\"SimplePagedResults\") \
(version 3.0; acl \"Anonymous simple paged results access\"; allow(read) \
userdn=\"ldap:///anyone\";)" \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--no-prompt
Replication problems
Replicas do not connect
If you set up servers with different deployment IDs, they cannot share encrypted data.
By default, they also cannot trust each other’s secure connections.
You may read messages like the following in the logs/errors
log file:
msg=Replication server accepted a connection from /address:port to local address /address:port but the SSL handshake failed.
Unless the servers use your own CA, make sure their keys are generated with the same deployment ID/password. Either set up the servers again with the same deployment ID, or refer to Replace deployment IDs.
Temporary delays
Replication can generally recover from conflicts and transient issues. Temporary delays are normal and expected while replicas converge, especially when the write load is heavy. This is a feature of eventual convergence, not a bug.
Persistently long replication delays can be a problem for client applications. A client application gets an unexpectedly old view of the data when reading from a very delayed replica. Monitor replication delay and take action when you observe persistently long delays. For example, make sure the network connections between DS servers are functioning normally. Make sure the DS server systems are sized appropriately.
For detailed suggestions about monitoring replication delays, refer to either of the following sections:
Use the logs
By default, replication records messages in the log file, logs/errors
.
Replication messages have category=SYNC
.
The messages have the following form. The following example message is folded for readability:
...msg=Replication server accepted a connection from 10.10.0.10/10.10.0.10:52859 to local address 0.0.0.0/0.0.0.0:8989 but the SSL handshake failed. This is probably benign, but may indicate a transient network outage or a misconfigured client application connecting to this replication server. The error was: Remote host closed connection during handshake
Stale data
DS replicas maintain historical information to bring replicas up to date and to resolve conflicts. To prevent historical information from growing without limit, DS replicas purge historical information after the replication-purge-delay (default: 3 days).
A replica becomes irrevocably out of sync when, for example:
-
You restore it from backup files older than the purge delay.
-
You stop it for longer than the purge delay.
-
The replica stays out of contact with other DS servers for longer than the purge delay.
When this happens to a single replica, reinitialize the replica.
For detailed suggestions about troubleshooting stale replicas, refer to either of the following sections:
Change number indexing
DS replication servers maintain a changelog database to record updates to directory data. The changelog database serves to:
-
Replicate changes, synchronizing data between replicas.
-
Let client applications get change notifications.
DS replication servers purge historical changelog data after the replication-purge-delay
in the same way replicas purge their historical data.
Client applications can get changelog notifications using cookies (recommended) or change numbers.
To support change numbers, the servers maintain a change number index to the replicated changes.
A replication server maintains the index when its configuration properties include
changelog-enabled:enabled
.
(Cookie-based notifications do not require a change number index.)
The change number indexer must not be interrupted for long. Interruptions can arise when, for example, a DS server:
-
Stays out of contact, not sending any updates or heartbeats.
-
Gets removed without being shut down cleanly.
-
Gets lost in a system crash.
Interruptions prevent the change number indexer from advancing. When a change number indexer cannot advance for almost as long as the purge delay, it may be unable to recover as the servers purge historical data needed to determine globally consistent change numbers.
For detailed suggestions about monitoring changelog indexing, refer to either of the following sections:
Take action based on the situation:
Situation | Actions to take |
---|---|
The time since last indexing is much smaller than the purge delay. |
No action required. |
The time since last indexing is approaching the purge delay. |
Begin by determining why. The fix depends on the exact symptoms. |
A DS server was removed without a clean shutdown. |
Rebuild an identical server and shut it down cleanly before removing it. |
A DS server disappeared in a crash. |
Rebuild an identical server. |
Incorrect configuration
When replication is configured incorrectly, fixing the problem can involve adjustments on multiple servers.
For example, adding or removing a bootstrap replication server means updating
the bootstrap-replication-server
settings in the synchronization provider configuration of other servers.
(The settings can be hard-coded in the configuration,
or read from the environment at startup time, as described in Property value substitution.
In either case, changing them involves at least restarting the other servers.)
For details, refer to Replication and the related pages.
Support
Sometimes you cannot resolve a problem yourself, and must ask for help or technical support. In such cases, identify the problem and how you reproduce it, and the version where you observe the problem:
$ status --offline --version
ForgeRock Directory Services 7.4.2-20240422170454-4479d6639a7cd2be7fa2765d7ad48f403369b3fc
Build <datestamp>
Be prepared to provide the following additional information:
-
The Java home set in
config/java.properties
. -
Access and error logs showing what the server was doing when the problem started occurring.
-
A copy of the server configuration file,
config/config.ldif
, in use when the problem started occurring. -
Other relevant logs or output, such as those from client applications experiencing the problem.
-
A description of the environment where the server is running, including system characteristics, hostnames, IP addresses, Java versions, storage characteristics, and network characteristics. This helps to understand the logs, and other information.
-
The
.zip
file generated using thesupportextract
command.For an example showing how to use the command, refer to supportextract.