The following steps demonstrate how to set up an HTTP port if none was configured at setup time with the --httpPort option:

The way directory data appears to client applications is configurable. You configure a Rest2ldap endpoint for each HTTP API to user data.

A Rest2ldap mapping file defines how JSON resources map to LDAP entries. The default mapping file is /path/to/opendj/config/rest2ldap/endpoints/api/example-v1.json. The default mapping works with Example.com data from the evaluation setup profile.

Edit or add mapping files for your own APIs. For details, refer to REST to LDAP reference.

If you have set up a directory server with the ds-evaluation profile, you can skip the first two steps:

HTTP authorization mechanisms map HTTP credentials to LDAP credentials.

Multiple HTTP authorization mechanisms can be enabled simultaneously. You can assign different mechanisms to Rest2ldap endpoints and to the Admin endpoint.

By default, these HTTP authorization mechanisms are supported:

When more than one authentication mechanism is specified, mechanisms are applied in the following order:

There are many possibilities when configuring HTTP authorization mechanisms. This procedure shows only one OAuth 2.0 example.

The example below uses settings as listed in the following table. When using secure connections, make sure the servers can trust each other’s certificates. Download ForgeRock Access Management software from the ForgeRock BackStage download site:

Read the ForgeRock Access Management documentation if necessary to install and configure AM. Then follow these steps to try the demonstration:

The APIs for configuring and monitoring DS servers are under the following endpoints:

To use the Admin endpoint APIs, follow these steps:

The reserved port number for LDAP is 389. Most examples in the documentation use 1389, which is accessible to non-privileged users:

StartTLS negotiations start on the unsecure LDAP port, and then protect communication with the client:

The reserved port number for LDAPS is 636. Most examples in the documentation use 1636, which is accessible to non-privileged users.

Directory schema, described in RFC 4512, define the kinds of information you find in the directory, and how the information is related.

By default, DS servers conform strictly to LDAPv3 standards for schema definitions and syntax checking. This ensures that data stored is valid and properly formed. Unless your data uses only standard schema present in the server when you install, you must add additional schema definitions to account for the data specific to your applications.

DS servers include many standard schema definitions. You can update and extend schema definitions while DS servers are online. As a result, you can add new applications requiring additional data without stopping your directory service.

The examples that follow focus primarily on the following types of directory schema definitions:

DS servers expose schema over protocol through the cn=schema entry. The server stores the schema definitions in LDIF format files in the db/schema/ directory. When you set up a server, the process copies many definitions to this location.

DS servers allow you to update directory schema definitions while the server is running. You can add support for new types of attributes and entries without interrupting the directory service. DS servers replicate schema definitions, propagating them to other replicas automatically.

You update schema either by:

Before adding schema definitions, take note of the following points:

For an example, refer to Custom schema.

DS servers support the standard LDAP schema definitions described in RFC 4512, section 4.1. They also support the following extensions:

Extensions to syntax definitions requires additional code to support syntax checking. DS servers support the following extensions for their particular use cases:

DS software has the following features for working with JSON objects:

DS servers treat JSON syntax attribute values as objects. Two JSON values may be considered equivalent despite differences in their string representations. The following JSON objects can be considered equivalent because each field has the same value. Their string representations are different, however:

Unlike other objects with their own LDAP attribute syntaxes, such as X.509 certificates, two JSON objects with completely different structures (different field names and types) are still both JSON. Nothing in the JSON syntax alone tells the server anything about what a JSON object must and may contain.

When defining LDAP schema for JSON attributes, it helps therefore to understand the structure of the expected JSON. Will the attribute values be arbitrary JSON objects, or JSON objects whose structure is governed by some common schema? If a JSON attribute value is an arbitrary object, you can do little to optimize how it is indexed or compared. If the value is a structured object, however, you can configure optimizations based on the structure.

For structured JSON objects, the definition of JSON object equality is what enables you to pick the optimal matching rule for the LDAP schema definition. The matching rule determines how the server indexes the attribute, and how the server compares two JSON values for equality. You can define equality in the following ways:

When you choose an equality matching rule in the LDAP attribute definition, you are also choosing the default that applies in an LDAP search filter equality assertion. For example, caseIgnoreJsonQueryMatch works with filters such as "(json=id eq 'bjensen')".

DS servers also implement JSON ordering matching rules for determining the relative order of two JSON values using a custom set of rules. You can select which JSON fields should be used for performing the ordering match. You can also define whether those fields that contain strings should be normalized before comparison by trimming white space or ignoring case differences. DS servers can implement JSON ordering matching rules on demand when presented with an extended server-side sort request, as described in Server-Side Sort. If, however, you define them statically in your LDAP schema, then you must implement them by creating a custom json-ordering-matching-rule Schema Provider. For details about the json-ordering-matching-rule object’s properties, refer to JSON Ordering Matching Rule.

For examples showing how to add LDAP schema for new attributes, refer to Update LDAP Schema. For examples showing how to index JSON attributes, refer to Custom Indexes for JSON.

By default, DS servers accept data that follows the schema for allowable and rejected data. You might have legacy data from a directory service that is more lenient, allowing non-standard constructions such as multiple structural object classes per entry, not checking attribute value syntax, or even not respecting schema definitions.

For example, when importing data with multiple structural object classes defined per entry, you can relax schema checking to warn rather than reject entries having this issue:

You can allow attribute values that do not respect the defined syntax:

You can even turn off schema checking altogether. Only turn off schema checking when you are absolutely sure that the entries and attributes already respect the schema definitions:

Turning off schema checking can potentially boost import performance.

DS servers provide many standard schema definitions. For full descriptions, refer to the Schema Reference. Find the definitions in LDIF files in the /path/to/opendj/db/schema/ directory:

The role of an index is to answer the question, "Which entries have an attribute with this corresponding value?"

Each index is therefore specific to an attribute.

Each index is also specific to the comparison implied in the search filter. For example, a directory server maintains distinct indexes for exact (equality) matching and for substring matching. The types of indexes are explained in Index types. Furthermore, indexes are configured in specific directory backends.

An index is implemented as a tree of key-value pairs. The key is a form of the value to match, such as babs jensen. The value is a list of IDs for entries that match the key. The figure that follows shows an equality (case ignore exact match) index with five keys from a total of four entries. If the data set were large, there could be more than one entry ID per key:

This example illustrates how DS uses an index.

When the search filter is (cn=Babs Jensen), DS retrieves the IDs for entries whose CN matches Babs Jensen by looking them up in the equality index of the CN attribute. (For a complex filter, it might optimize the search by changing the order in which it uses the indexes.) A successful result is zero or more entry IDs. These are the candidate result entries.

For each candidate, DS retrieves the entry by ID from a system index called id2entry. As its name suggests, this index returns an entry for an entry ID. If there is a match, and the client application has the right to access the data, DS returns the search result. It continues this process until no candidates are left.

If there are no indexes that correspond to a search request, DS must check for a match against every entry in the scope of the search. Evaluating every entry for a match is referred to as an unindexed search.

An unindexed search is an expensive operation, particularly for large directories. A server refuses unindexed searches unless the user has specific permission to make such requests. The permission to perform an unindexed search is granted with the unindexed-search privilege. This privilege is reserved for the directory superuser by default. It should not be granted lightly.

If the number of entries is smaller than the default resource limits, you can still perform what appear to be unindexed searches, meaning searches with filters for which no index appears to exist. That is because the dn2id index returns all user data entries without hitting a resource limit that would make the search unindexed.

Use cases that may call for unindexed searches include the following:

When an entry is added, changed, or deleted, the directory server updates each affected index to reflect the change. This happens while the server is online, and has a cost. This cost is the reason to maintain only those indexes that are used.

DS only updates indexes for the attributes that change. Updating an unindexed attribute is therefore faster than updating an indexed attribute.

A presence index matches an attribute that is present on the entry, regardless of the value. By default, the aci attribute is indexed for presence:

A presence index takes up less space than other indexes. In a presence index, there is just one key with a list of IDs.

The following command examines the ACI presence index for a server configured with the evaluation profile:

In this case, entries with ACI attributes have IDs 1 and 9.

An equality index matches values that correspond exactly (generally ignoring case) to those in search filters. An equality index requires clients to match values without wildcards or misspellings:

An equality index has one list of entry IDs for each attribute value. Depending on the backend implementation, the keys in a case-insensitive index might not be strings. For example, a key of 6A656E73656E could represent jensen.

The following command examines the SN equality index for a server configured with the evaluation profile:

In this case, there are 17 entries that have an SN of Jensen.

Unless the keys are encrypted, the server can reuse an equality index for ordering and initial substring searches.

An approximate index matches values that "sound like" those provided in the filter. An approximate index on sn lets client applications find people even when they misspell surnames:

An approximate index squashes attribute values into a normalized form.

The following command examines an SN approximate index added to a server configured with the evaluation profile:

In this case, there are 74 entries that have an SN that sounds like Jensen.

A substring index matches values that are specified with wildcards in the filter. Substring indexes can be expensive to maintain, especially for large attribute values:

In a substring index, there are enough keys to match any substring in the attribute values. Each key is associated with a list of IDs. The default maximum size of a substring key is 6 bytes.

The following command examines an SN substring index for a server configured with the evaluation profile:

In this case, the SN value Jensen shares substrings with many other entries. The size of the lists and number of keys make a substring index much more expensive to maintain than other indexes. This is particularly true for longer attribute values.

An ordering index is used to match values for a filter that specifies a range. For example, the ds-sync-hist attribute used by replication has an ordering index by default. Searches on that attribute often seek entries with changes more recent than the last time a search was performed.

The following example shows a search that specifies a range on the SN attribute value:

In this case, the server only requires an ordering index if it cannot reuse the (ordered) equality index instead. For example, if the equality index is encrypted, an ordering index must be maintained separately.

A big index is designed for attributes where many, many entries have the same attribute value.

This can happen for attributes whose values all belong to a known enumeration. For example, if you have a directory service with an entry for each person in the United States, the st (state) attribute is the same for more than 30 million Californians (st: CA). With a regular equality index, a search for (st=CA) would be unindexed. With a big index, the search is indexed, and optimized for paging through the results. For an example, refer to Indexes for attributes with few unique values.

A big index can be easier to configure and to use than a virtual list view index. Consider big indexes when:

¹ Many, but not all entries. Do not create a big index for all values of the objectClass attribute, for example. When all entries have the same value for an attribute, as is the case for objectClass: top, indexes consume additional system resources and disk space with no benefit. The DS server must still read every entry to return search results. In practice, the upper limit is probably somewhat less than half the total entries. In other words, if half the entries have the same value for an attribute, it will cost more to maintain the big index than to evaluate all entries to find matches for the search. Let such searches remain unindexed searches.

A virtual list view (VLV) or browsing index is designed to help applications that list results. For example, a GUI application might let users browse through a list of users. VLVs help the server respond to clients that request server-side sorting of the search results.

VLV indexes correspond to particular searches. Configure your VLV indexes using the command line.

In some cases, you need an index for a matching rule other than those described above.

For example, a generalized time-based matching index lets applications find entries with a time-based attribute later or earlier than a specified time.

Index maintenance has its costs. Every time an indexed attribute is updated, the server must update each affected index to reflect the change. This is wasteful if the index is not used. Indexes, especially substring indexes, can occupy more memory and disk space than the corresponding data.

Aim to maintain only indexes that speed up appropriate searches, and that allow the server to operate properly. The former indexes depend on how directory users search, and require thought and investigation. The latter includes non-configurable internal indexes, that should not change.

Begin by reviewing the attributes of your directory data. Which attributes would you expect to find in a search filter? If an attribute is going to show up frequently in reasonable search filters, then index it.

⁽¹⁾ DS servers optimize the search internally to:

Compare your guesses with what you observe actually happening in the directory. For more complex filters, refer to Debug search indexes.

Directory users might complain their searches fail because they’re unindexed.

Ask for the result code, additional information, and search filter. By default, DS directory servers reject unindexed searches with a result code of 50 and additional information about the unindexed search. The following example attempts, anonymously, to get the entries for all users whose email address ends in .com:

If they’re unintentionally requesting an unindexed search, suggest ways to perform an indexed search instead. Perhaps the application needs a better search filter. Perhaps it requests more results than necessary. For example, a GUI application lets a user browse directory entries. The application could page through the results, rather than attempting to retrieve all the entries at once. To let the user page back and forth through the results, you could add a browsing (VLV) index for the application to get the entries for the current screen.

An application might have a good reason to get the full list of all entries in one operation. If so, assign the application’s account the unindexed-search privilege. Consider other options before you grant the privilege, however. Unindexed searches cause performance problems for concurrent directory operations.

When an application has the privilege or binds with directory superuser credentials—​by default, the uid=admin DN and password—​then DS does not reject its request for an unindexed search. Check for unindexed searches using the ds-mon-backend-filter-unindexed monitoring attribute:

If ds-mon-backend-filter-unindexed is greater than zero, review the access log for unexpected unindexed searches. The following example shows the relevant fields in an access log message:

Beyond the key fields shown in the example, messages in the access log also specify the search filter and scope. Understand the operation that led to each unindexed search. If the filter is appropriate and often used, add an index to ease the search. Either analyze the access logs to find how often operations use the search filter or monitor operations with the index analysis feature, described in Index analysis metrics.

In addition to responding to client search requests, a server performs internal searches. Internal searches let the server retrieve data needed for a request, and maintain internal state information. Sometimes, internal searches become unindexed. When this happens, the server logs a warning similar to the following:

When you observe a message like this in the server log, take these actions:

DS servers provide the index analysis feature to collect information about filters in search requests. This feature is useful, but not recommended to keep enabled on production servers, as DS maintains the metrics in memory.

You can activate the index analysis mechanism using the dsconfig set-backend-prop command:

The command causes the server to analyze filters used, and to keep the results in memory. You can read the results as monitoring information:

The ds-mon-backend-filter-indexed and ds-mon-backend-filter-unindexed metrics are available even when index filter analysis is disabled.

The ds-mon-backend-filter-use values include the following fields:

The output can include filters for internal use, such as (aci=*). In the example above, you observe a filter used by a client application.

In the example, a search filter that led to an unindexed search, (employeenumber=86182), had no matches because, "caseIgnoreMatch index type is disabled for the employeeNumber attribute". Some client application has tried to find users by employee number, but no index exists for that purpose. If this appears regularly as a frequent search, add an employee number index.

To avoid impacting server performance, turn off index analysis after you collect the information you need. Use the dsconfig set-backend-prop command:

Sometimes it is not obvious by inspection how a directory server processes a given search request. The directory superuser can gain insight with the debugsearchindex attribute.

The default global access control only allows the directory superuser to read the debugsearchindex attribute. To allow another user to read the attribute, add a global ACI such as the following:

The debugsearchindex attribute value indicates how the server would process the search. The server uses its indexes to prepare a set of candidate entries. It iterates through the set to compare candidates with the search filter, returning entries that match. The following example demonstrates this feature for a subtree search with a complex filter:

The filter in the example matches person entries whose given name starts with aa. The search scope is not explicitly specified, so the scope defaults to the subtree including the base DN.

Notice that the debugsearchindex value has the following top-level fields:

In the output, notice that the server uses the equality and substring indexes to find candidate entries whose given name starts with aa. If the filter indicated given names containing aa, as in givenName=*aa*, the server would rely only on the substring index.

Notice that the output for the (objectclass=person) portion of the filter shows "candidates": "[LIMIT-EXCEEDED]". In this case, there are so many entries matching the value specified that the index is not useful for narrowing the set of candidates. The scope is also not useful for narrowing the set of candidates. Ultimately, however, the givenName indexes help the server to narrow the set of candidates. The overall search is indexed and the result is 50 matching entries.

If an index already exists, but you suspect it is not working properly, refer to Verify indexes.

DS maintains metrics about index use. The metrics indicate how often an index was accessed since the DS server started.

The following examples demonstrate how to read the metrics for all monitored indexes:

If the number of attribute index uses is persistently zero, then you can eventually conclude the index is unused. Of course, it is possible that an index is needed, but has not been used since the last server restart. Be sure to sample often enough that you know the index is unused before taking action.

You can remove unused attribute indexes with one of the following commands, depending on the type of index:

DS maintains metrics about index cost, which lets you determine which indexes are causing write contention. The metrics count the number of updates and how long they took since the DS server started.

The following examples demonstrate how to read the metrics for all monitored indexes:

If the cost is persistently very high, and an attribute index is not used, or hardly used, then consider removing it. Never remove a system index.

Big indexes fit the case where many, many entries have the same attribute value.

The default DS evaluation profile generates 100,000 user entries with addresses in the United States, so some st (state) attributes are shared by 4000 or more users. With a regular equality index, searches for some states reach the index entry limit, causing unindexed searches. A big index avoids this problem.

The following commands configure an equality index for the state attribute, and then build the new index:

Once the index is ready, a client application can page through all the users in a state with an indexed search:

When creating a big index that uses an extensible matching rule, model your work on the example in Extensible match index, but use the following options to set the index type and the matching rule:

DS servers support attribute values that have JSON syntax. The following schema excerpt defines a json attribute with case-insensitive matching:

When you index a JSON attribute defined in this way, the default directory server behavior is to maintain index keys for each JSON field. Large or numerous JSON objects can result in large indexes, which is wasteful. If you know which fields are used in search filters, you can choose to index only those fields.

As described in Schema and JSON, for some JSON objects only a certain field or fields matter when comparing for equality. In these special cases, the server can ignore other fields when checking equality during updates, and you would not maintain indexes for other fields.

How you index a JSON attribute depends on the matching rule in the attribute’s schema definition, and on the JSON fields you expect to be used as search keys for the attribute.

The following example shows how to create a VLV index. This example applies where GUI users browse user accounts, sorting on surname then given name:

When you first import directory data, the directory server builds the indexes as part of the import process. DS servers maintain indexes automatically, updating them as directory data changes.

Only rebuild an index manually when it is necessary to do so. Rebuilding valid indexes wastes server resources, and is disruptive for client applications.

However, you must manually intervene when you:

An index is a tree of key-value pairs. The key is what the search is trying to match. The value is a list of entry IDs.

As the number of entries in the directory grows, the list of entry IDs for some keys can become very large. For example, every entry in the directory has objectClass: top. If the directory maintains a substring index for mail, the number of entries ending in .com could be huge.

A directory server therefore defines an index entry limit. When the number of entry IDs for a key exceeds the limit, the server stops maintaining a list of IDs for that key. The limit effectively means a search using only that key is unindexed. Searches using other keys in the same index are not affected.

The following figure shows a fragment from a substring index for the mail attribute. The number of email addresses ending in com has exceeded the index entry limit. For the other substring keys, the entry ID lists are still maintained. To save space, the entry IDs are not shown in the figure.

Ideally, the limit is set at the point where it becomes more expensive to maintain the entry ID list for a key, and to perform an indexed search than to perform an unindexed search. In practice, the limit is a tradeoff, with a default index entry limit value of 4000. Keep the default setting unless you have good reason to change it.

In this example, an LDAP client application helps people look up names using mobile telephone numbers. The mobile numbers are stored on the mobile attribute in the directory.

The LDAP client application has a search for a mobile number failing with error 50:

The application owner tells you there’s a problem searching on mobile numbers.

As administrator, you can observe the failures in the DS access logs. The following example includes only the relevant fields of an access log message with the failure:

For this simple filter, (mobile=14120300042), if the search is uninindexed, you can conclude that the attribute must not be indexed. As expected, the mobile attribute does not appear in the list of indexes for the backend:

If the filter were more complex, you could run the search with the debugsearchindex attribute to determine why it is unindexed.

You notice that telephoneNumber has equality and substring indexes, and decide to add the same for the mobile attribute. Adding a new index means adding the configuration for the index, and then building the index. An index is specific to a given server, so you do this for each DS replica. For example:

Once the index is built, you check that the search is now indexed by looking at the debugsearchindex output:

You tell the application owner to try searching on mobile numbers now that you have indexed the attribute.

The client application now has a working search:

Must you do anything more? What about index-entry-limit settings, for example?

Stop the server, and run the backendstat show-index-status command:

Notice that the equality index mobile.telephoneNumberMatch has no keys whose entry ID lists are over the limit, and the average number of values is 1. This is what you would expect. Each mobile number belongs to a single user. DS uses this index for exact match search filters like (mobile=14120300042).

Notice also that the substring index mobile.telephoneNumberSubstringsMatch:6 has 10 keys whose lists are over the (default) limit of 4000 values. These are the keys for substrings that match only a single digit of a mobile number. For example, a search for all users whose mobile telephone number starts with 1 uses the filter (mobile=1*). This search would be unindexed.

Should you raise the index-entry-limit for this substring index? Probably not, no.

The filter (mobile=1*) matches all mobile numbers for the United States and Canada, for example. Someone running this search is not looking up a user’s name by their mobile phone number. They are scanning the directory database, even if it is not intentional. If you raise the index-entry-limit setting to prevent any Over index-entry-limit keys, the server must update enormous entry ID lists for these keys whenever a mobile attribute value changes. The impact on write performance could be a bad tradeoff.

If possible, suggest that the LDAP application refrains from searching until the user has provided enough digits of the mobile number to match, or that it prompts the user for more digits when it encounters an unindexed search.

If you cannot change the application, it might be acceptable that searches for a single mobile telephone digit simply fail. That might be a better tradeoff than impacting write performance due to a very high index-entry-limit setting.

When you create a new backend on a replicated directory server, let the server replicate the new data:

If you must temporarily disable replication for the backend, take care to avoid losing changes. For details, refer to Disable replication.

The following procedures demonstrate how to import and export LDIF data.

For details on creating custom sample LDIF to import, refer to Generate test data.

Splitting data impacts replication, and can require that you interrupt the service.

On one set of replicas, create the exampleData backend to hold all dc=example,dc=com data except ou=People,dc=example,dc=com:

On another set of replicas, create a peopleData backend to hold ou=People,dc=example,dc=com data:

After completing the data import process, start the replicas.

To split an existing backend, follow these high-level steps:

Directory data growth depends on applications that use the directory. When directory applications add more data than they delete, the database backend grows until it fills the available disk space. The system can end up in an unrecoverable state if no disk space is available.

Database backends therefore have advanced properties, disk-low-threshold and disk-full-threshold. When available disk space falls below disk-low-threshold, the directory server only allows updates from users and applications that have the bypass-lockdown privilege. When available space falls below disk-full-threshold, the directory server stops allowing updates, instead returning an UNWILLING_TO_PERFORM error to each update request.

If growth across the directory service tends to happen quickly, set the thresholds higher than the defaults to allow more time to react when growth threatens to fill the disk. The following example sets disk-low-threshold to 10 GB disk-full-threshold to 5 GB for the dsEvaluation backend:

The properties disk-low-threshold, and disk-full-threshold are listed as advanced properties.

To examine their values with the dsconfig command, use the --advanced option:

If the directory service creates many entries that expire and should be deleted, you could find the entries with a time-based search and then delete them individually. That approach uses replication to delete the entries in all replicas. It has the disadvantage of generating potentially large amounts of replication traffic.

Entry expiration lets you configure the backend database to delete the entries as they expire. This approach deletes expired entries at the backend database level, without generating replication traffic. AM uses this approach when relying on DS to delete expired token entries, as demonstrated in Install DS for AM CTS.

Backend indexes for generalized time (timestamp) attributes have these properties to configure automated, optimized entry expiration and removal:

Configure this capability by performing the following steps:

Once you configure and build the index, the backend can delete expired entries. At intervals of 10 seconds, the backend automatically deletes entries whose timestamps on the attribute are older than the specified lifetime. Entries that expire in the interval between deletions are removed on the next round. Client applications should therefore check that entries have not expired, as it is possible for expired entries to remain available until the next round of deletions.

When using this capability, keep the following points in mind:

The following example adds the base DN o=example to the dsEvaluation backend, and creates a replication domain configuration to replicate the data under the new base DN:

When you add a base DN to a backend, the base DN must not be subordinate to any other base DNs in the backend. As in the commands shown here, you can add the base DN o=example to a backend that already has a base DN dc=example,dc=com. You cannot, however, add o=example,dc=example,dc=com as a base DN, because that is a child of dc=example,dc=com.

When you delete a database backend with the dsconfig delete-backend command, the directory server does not actually remove the database files for these reasons:

After you run the dsconfig delete-backend command, manually remove the database backend files, and remove replication domain configurations any base DNs you deleted by removing the backend.

If run the dsconfig delete-backend command by mistake, but have not yet deleted the actual files, recover the data by creating the backend again, and reconfiguring and rebuilding the indexes.

A dynamic group references members using LDAP URLs. Dynamic group entries take the groupOfURLs object class, with one or more memberURL values specifying LDAP URLs to identify group members.

Dynamic groups are a natural fit for directory servers. If you have a choice, choose dynamic groups over static groups for these reasons:

The following dynamic group includes entries matching the filter "(l=San Francisco)" (users whose location is San Francisco):

Group membership changes dynamically as entries change to match the memberURL values:

DS servers let you create virtual static groups. Virtual static groups allow applications to display dynamic groups as if they had an enumerated list of members like a static group.

The virtual static group takes the auxiliary object class ds-virtual-static-group. Virtual static groups use the object class groupOfNames, or groupOfUniqueNames. Instead of member or uniqueMember attributes, they have ds-target-group-dn attributes pointing to other groups.

Generating the list of members can be resource-intensive for large groups. By default, you cannot retrieve the list of members. If you have an application that must read the list of members, change the configuration of Virtual Static member or Virtual Static uniqueMember to set allow-retrieving-membership:true:

The following example creates a virtual static group, and reads the group entry with all members:

A static group entry enumerates the entries in the group. Static group entries grow as their membership increases.

Large static groups are a performance bottleneck. If you have a choice, choose dynamic groups over static groups for these reasons:

Static group entries are based on one of the following standard object classes:

Like other LDAP attributes, each group member attribute value is unique. LDAP does not allow duplicate values for the same attribute on the same entry.

To create a static group, add a group entry such as the following to the directory:

To change group membership, modify the values of the membership attribute:

RFC 4519 says a groupOfNames entry must have at least one member. Although DS servers allow you to create a groupOfNames without members, strictly speaking, that behavior is not standard. Alternatively, you can use the groupOfEntries object class as shown in the following example:

DS servers let you nest groups. The following example shows a group of groups of managers and administrators:

Although not shown in the example above, DS servers let you nest groups within nested groups.

DS servers let you create dynamic groups of groups. The following example shows a group of other groups. The members of this group are themselves groups, not users:

Use the isMemberOf attribute to determine what groups a member belongs to, as described in Group membership. The following example requests the groups that Kirsten Vaughan belongs to:

Notice that Kirsten is a member of The Big Shots group.

Notice also that Kirsten does not belong to the Group of Groups. The members of that group are groups, not users. The following example requests the groups that the directory administrators group belongs to:

The following example shows which groups each group belong to. The search is unindexed, and so is performed here with directory administrator credentials:

Notice that the group of groups is not a member of itself.

DS servers let you verify which groups a user belongs to by reading their entry. The virtual isMemberOf attribute shows which groups a user is in.

Reading the user entry is more efficient than reading large static group entries and checking the lists of members:

You must request isMemberOf explicitly.

For a dynamic group, you can check the membership directly on the candidate member’s entry using the same search criteria as the dynamic group’s member URL.

When you delete or rename an entry that belongs to static groups, that entry’s DN must be removed or changed in each group it belongs to. You can configure the server to resolve membership changes by enabling referential integrity.

Referential integrity functionality is implemented as a plugin. The referential integrity plugin is disabled by default. To enable the plugin, use the dsconfig command:

With the plugin enabled, referential integrity resolves group membership automatically:

By default, the referential integrity plugin is configured to manage member and uniqueMember attributes. These attributes take values that are DNs, and are indexed for equality by default for the default backend. Before you add an additional attribute to manage, make sure that it has DN syntax and that it is indexed for equality. DS servers require indexes because an unindexed search can consume too many of the server’s resources. For instructions on indexing attributes, refer to Configure indexes.

Consider these settings when configuring the referential integrity plugin:

If you must use the entryDN and isMemberOf virtual attributes in a filter, use a simple equality filter. The following example shows how to add access to read isMemberOf, and then run a search that returns the common names for members of a group:

Virtual attributes are generally operational, and so, are returned only when explicitly requested. The searches in these examples are unindexed, are therefore performed with directory administrator credentials:

DS servers define the following virtual attributes by default:

You can use the existing virtual attribute types to create your own virtual attributes. You can also use the user-defined type to create your own virtual attribute types. The virtual attribute is defined by the server configuration, which is not replicated:

DS lets you create virtual attributes based on the values of non-virtual attributes. The following example overrides the mail attribute on user entries with a user-template virtual attribute:

A template string, such as {givenName}.{sn}@{domain|virtual.example.com}, follows these rules:

You can read a virtual LDAP attribute over REST as you would any other attribute:

This example defines attributes that depend on the user’s service level.

This example shows collective attributes that depend on a classOfService attribute value:

You define collective attributes in the user data using an LDAP subentry. As a result, collective attribute settings are replicated. Collective attributes use attributes defined in the directory schema. The following LDIF shows the schema definitions:

The collective attribute definitions set the quotas depending on class of service:

When the collective attributes are defined, you observe the results on user entries. Before trying this example, set up a directory server with the ds-evaluation profile:

This example demonstrates how to set an employee’s department number using the manager’s department number.

The employee-manager relationship is defined by the employee’s manager attribute. Each manager attribute specifies the DN of a manager’s entry. The server looks up the manager’s department number to set the attribute on the employee’s entry.

The collective attribute subentry specifies that users inherit department number from their manager:

Babs Jensen’s manager is Torrey Rigden:

Torrey’s department number is 3001:

Babs inherits her department number from Torrey. Before trying this example, set up a directory server with the ds-evaluation profile:

This example demonstrates how to set a user’s default language preferences and street address based on locality.

For this example, the relationship between entries is based on locality. The collective attribute subentry specifies how to construct the RDN of the entry with the values to inherit:

The RDN of the entry from which to inherit attributes is l=localityName,ou=Locations, where localityName is the value of the l (localityName) attribute on the user’s entry.

In other words, if the user’s entry has l: Bristol, then the RDN of the entry from which to inherit attributes starts with l=Bristol,ou=Locations. The actual entry looks like this:

The subentry specifies that the inherited attributes are preferred language and street address.

The object class extensibleObject allows the entry to take a preferred language. The object class extensibleObject means, "Let me add whatever attributes I want." The best practice is to add a custom auxiliary object class instead. The example uses shortcut extensibleObject for simplicity.

Notice the last line of the collective attribute subentry:

This line indicates that when a collective attribute clashes with a real attribute, the real value takes precedence over the virtual, collective value. You can set collectiveConflictBehavior to virtual-overrides-real for the opposite precedence, or to merge-real-and-virtual to keep both sets of values.

In this example, users can set their own language preferences. When users set language preferences manually, the user’s settings override the locality-based setting.

Sam Carter is located in Bristol. Sam has specified no preferred languages:

Sam inherits the street address and preferred language from the Bristol locality. Before trying this example, set up a directory server with the ds-evaluation profile:

Babs’s locality is San Francisco. Babs prefers English, but also knows Korean:

Babs inherits the street address from the San Francisco locality, but keeps her language preferences:

This example demonstrates how to inherit a description from the parent entry.

The following collective attribute subentry specifies that entries inherit a description from their parent unless they already have a description:

In this example, inheritFromDNAttribute uses the virtual attribute entryDN. The setting inheritFromDNParent: 1 indicates that the server should locate the parent entry by removing one leading RDN from the entryDN. (To inherit from the grandparent entry, you would specify inheritFromDNParent: 2, for example.)

Sam Carter’s entry has no description attribute, and so inherits the parent’s description:

Babs Jensen’s entry already has a description attribute, and so does not inherit from the parent:

LDAP subentries reside with the user data and so the server replicates them. Subentries hold operational data. They are not visible in search results unless explicitly requested. This section describes how a subentry’s subtreeSpecification attribute defines the scope of the subtree that the subentry applies to.

An LDAP subentry’s subtree specification identifies a subset of entries in a branch of the DIT. The subentry scope is these entries. In other words, these are the entries that the subentry affects.

The attribute value for a subtreeSpecification optionally includes the following parameters:

The following illustration shows this for an example collective attribute subentry:

Notice that the base of ou=People on the subentry cn=Silver Class of Service,dc=example,dc=com indicates that the base entry is ou=People,dc=example,dc=com.

The filter "(classOfService=silver)" means that Kirsten Vaughan and Sam Carter’s entries are in scope. Babs Jensen’s entry, with classOfService: bronze does not match and is therefore not in scope. The ou=People organizational unit entry does not have a classOfService attribute, and so is not in scope, either.

When you set up a server, you can specify the following:

Setup profiles that create backends for schema and directory data configure replication domains for their base DNs. The server is ready to replicate that directory data when it starts up.

Replication initialization depends on the state of the data in the replicas.

DS replication shares changes, not data. When a replica applies an update, it sends a message to its replication server. The replication server forwards the update to all other replication servers, and to its replicas. The other replication servers do the same, so the update is eventually propagated to all replicas.

Each replica eventually converges on the same data by applying each update and resolving conflicts in the same way. As long as each replica starts from the same initial state, each replica eventually converges on the same state. It is crucial, therefore, for each replica to begin in the same initial state. Replicas cannot converge by following exactly the same steps from different initial states.

Internally, DS replicas store a shorthand form of the initial state called a generation ID. The generation ID is a hash of the first 1000 entries in a backend. If the replicas' generation IDs match, the servers can replicate data without user intervention. If the replicas' generation IDs do not match for a given backend, you must manually initialize replication between them to force the same initial state on all replicas.

If necessary, and before starting the servers, further restrict TLS protocols and cipher suites on all servers. This forces the server to use only the restricted set of protocols and cipher suites. For details, refer to TLS settings.

The primary unit of replication is the replication domain. A replication domain has a base DN, such as dc=example,dc=com.

The set of DS replicas and replication servers sharing one or more replication domains is a replication topology. Replication among these servers applies to all the data under the domain’s base DN.

The following example topology replicates dc=example,dc=com, dc=example,dc=org, and dc=example,dc=net. All the replication servers in a topology are fully connected, replicating the same data under each base DN:

Replication doesn’t support separate, independent domains for the same base DN in the same topology. For example, you can’t replicate two dc=example,dc=org domains with different data in the same topology:

When you set up a replication domain, replicate the data under the base DN among all the servers in the topology. If the data under a base DN is different on different servers, configure the servers appropriately:

Replication depends on the directory schema under cn=schema. If applications require specific, non-standard schema definitions or can update the LDAP schema online, replicate cn=schema with the other base DNs.

DS servers listen on dedicated ports for administrative requests and for replication requests. These dedicated ports must remain open to remote requests from configuration tools and from other servers. Make sure that firewall software allows connections to the administration and replication ports from all connecting servers.

DS server configuration tools securely connect to administration ports. Administrative connections are short-lived.

DS replicas connect to DS replication ports for replication requests. A server listening on a replication port is called a replication server, whether it is running inside the same process as a directory server, or on a separate host system. Replication connections are long-lived. Each DS replica connects to a replication server upon initialization and then at startup time. The replica keeps the connection open to push and receive updates, although it can connect to another replication server.

The command to initialize replication uses the administrative port, and initialization uses the replication port:

DS replicas push updates to and receive updates from replication servers over replication connections. When processing an update, a replica (DS) pushes it to the replication server (RS) it is connected to. The replication server pushes the update to connected replicas and to other replication servers. Replicas always connect through replication servers.

A replica with a replication port and a changelog plays both roles (DS/RS), normally connecting to its own replication server. A standalone replica (DS) connects to a remote replication server (RS). The replication servers connect to each other. The following figure shows the flow of messages between standalone replicas and replication servers.

The command to monitor replication status uses the administration ports on multiple servers to connect and read monitoring information, as shown in the following sequence diagram:

DS servers can provide both directory services and replication services. The two services are not the same, even if they usually run alongside each other in the same DS server.

Replication relies on the replication service provided by DS replication servers. DS directory servers (replicas) publish changes made to their data, and subscribe to changes published by other replicas. The replication service manages replication data only, sending and receiving replication messages. A replication server receives, sends, and stores only changes to directory data, not the data itself.

The directory service manages directory data. It responds to requests, and stores directory data and historical information. For each replicated base DN, such as dc=example,dc=com or cn=schema, the directory service publishes changes to and subscribes to changes from a replication service. The directory service resolves any conflicts that arise when reconciling changes from other replicas, using the historical information about changes to resolve the conflicts. (Conflict resolution is the responsibility of the directory server rather than the replication server.)

After a directory server connects to a replication topology, it connects to one replication server at a time for a given domain. The replication server provides the directory server with the list of all replication servers for that base DN. Given this list, the directory server selects its preferred replication server when starting up, when it loses the current connection, or when the connection becomes unresponsive.

For each replicated base DN, a directory server prefers to connect to a replication server:

Consider a dc=example,dc=com topology with five directory servers connected to replication servers A, B, and C, where:

Replication server C is the server with the most available capacity. All other criteria being equal, replication server C is the server to connect to when another directory server joins the topology.

The directory server regularly updates the list of replication servers in case it must reconnect. As available capacity can change dynamically, a directory server can reconnect to another replication server to balance the replication load in the topology. For this reason, the server can also end up connected to different replication servers for different base DNs.

Review Initialization options before following these steps:

Review Initialization options before following these steps:

Initialize each replica with the same LDIF:

Review Initialization options before following these steps:

After you add a replication server to a deployment, add it to the other servers' bootstrap-replication-server settings.

Apply these steps for each server whose configuration references the new replication server to add:

After you remove a replication server from a deployment, remove it from other servers' bootstrap-replication-server settings.

Apply these steps for each server whose configuration references the replication server that you removed:

Follow these steps to disable replication temporarily for a replica and drop any changes that occur:

Before removing a server from a group of replicated servers, disable replication as described. When the server you remove is a bootstrap replication server, also remove it from the configuration on all other servers.

You might remove a server from a replication topology because:

To remove a server that is no longer needed:

To change which servers replicate with each other:

DS servers that have a replication server port and an LDAP port publish an external changelog over LDAP:

DS servers do not encrypt changelog data on disk by default. Any user with system access to read directory files can potentially read external changelog data.

In addition to preventing read access by other users, you can configure confidentiality for changelog data. When confidentiality is enabled, the server encrypts changelog records before storing them on disk. The server decrypts changelog records before returning them to client applications.

DS servers encrypt data using symmetric keys. Servers store symmetric keys, encrypted with the shared master key, with the data they encrypt. As long as servers have the same shared master key, any server can recover symmetric keys needed to decrypt data.

Symmetric keys for (deprecated) reversible password storage schemes are the exception to this rule. When you configure a reversible password storage scheme, enable the adminRoot backend, and configure a replication domain for cn=admin data.

Encrypting and decrypting data require cryptographic processing that reduces throughput, and extra space for larger encrypted values. Tests with default settings show that the cost of enabling confidentiality can be quite modest. Your results can vary based on the host system hardware, the JVM, and the settings for cipher-transformation and cipher-key-length. Make sure you test your deployment to qualify the impact of confidentiality before changing settings in production.

For a user to read the changelog, the user must have access to read, search, and compare changelog attributes, might have access to use the control to read the external changelog, and must have the changelog-read privilege.

The changes returned from a search on the external changelog include only what was actually changed. If you have applications that need additional attributes published with every changelog entry, regardless of whether the attribute itself has changed, specify those with the ecl-include and ecl-include-for-deletes properties:

Exclude domains to prevent applications that read the external changelog from having to process update notifications for entries that are not relevant to them:

By default, a replication server indexes change numbers for replicated user data. This allows applications to get update notifications by change number. Indexing change numbers requires additional CPU, disk I/O, and storage. Disable it if none of your applications require change number-based browsing:

Suppose the entries below ou=Subscribers,dc=example,dc=com are stored on a different directory server at referral.example.com:1636. You can create a LDAP referral to reference the remote entries.

Before creating the LDAP referral, give directory administrators access to use the Manage DSAIT control, and to manage the ref attribute:

To create an LDAP referral, either create a referral entry, or add the extensibleObject object class and the ref attribute with an LDAP URL to an existing entry. The example above creates a referral entry at ou=Subscribers,dc=example,dc=com:

DS servers can now return a referral for operations under ou=Subscribers:

To access the entry instead of the referral, use the Manage DSAIT control:

You can use the Manage DSAIT control to change the referral with the ldapmodify command:

By default, the unique attribute plugin is configured to ensure unique uid values:

You can configure the unique attribute plugin for use with any attributes, not just uid:

In some cases, attributes must be unique, but only in the context of a particular base DN. For example, uid values must be unique under dc=example,dc=com and under dc=example,dc=org. But it is okay to have uid=bjensen,ou=people,dc=example,dc=com and uid=bjensen,ou=people,dc=example,dc=org:

The unique attribute plugin only ensures uniqueness on the replica where the attribute is updated. If client applications write the same attribute value separately at the same time on different replicas, both replicas might use the same "unique" value, especially if the network is down between the replicas:

The Samba Administrator updates the LDAP password when a Samba password changes.

In Samba’s smb.conf configuration file, the value of ldap admin dn is set to the DN of this account. When the Samba Administrator changes a user password, the plugin ignores the changes. Choose a distinct account different from the directory superuser and other administrators:

When the target DN of an LDAP request is not in a local backend, an LDAP server can refuse to handle the request, or return a referral. An LDAP proxy can also forward the request to another directory server.

In Directory Services, the LDAP proxy is implemented as a proxy backend. Rather than store user data locally, the proxy backend forwards requests to remote directory servers.

For proxy backends, a service discovery mechanism identifies remote directory servers to forward LDAP requests to. A service discovery mechanism’s configuration specifies the keys used for secure communications, and how to contact the remote directory servers. It reads remote directory servers' configurations to discover their capabilities, including the naming contexts they serve, and so which target DNs they can handle. It periodically rereads their configurations in case they have been updated since the last service discovery operation.

When preparing to configure a service discovery mechanism, choose one of these alternatives:

When configuring a service discovery mechanism, make sure all the remote directory servers are replicas of each other, and that they have the same capabilities. A proxy backend expects all remote directory server replicas known to the mechanism to hold the same data. This allows the backend to treat the replicas as equivalent members of a pool. In the configuration, a pool of equivalent replicas is a shard.

In deployments where you must distribute data for horizontal write scalability, you can configure multiple service discovery mechanisms. The proxy backend can then distribute write requests across multiple shards.

The following example creates a replication service discovery mechanism that specifies two replication servers:

The example above assumes that the servers protect connections using keys generated with a deployment ID and password. If this is not the case, configure the security settings appropriately.

The following example creates a static service discovery mechanism that specifies four remote LDAP servers:

The example above assumes that the remote servers use certificates signed by well-known CAs, and so are recognized using the trust manager provided by the JVM. If this is not the case, configure the security settings appropriately.

If the proxy server must perform SSL mutual authentication when setting up secure connections with the remote servers, configure an appropriate key manager provider and SSL certificate nickname.

Supported service discovery mechanisms define a discovery-interval that specifies how often to read the remote server configurations to discover changes. Because the mechanism polls periodically for configuration changes, by default, it can take up to one minute for the mechanism to observe the changes. If necessary, you can change this setting in the configuration.

A proxy backend forwards requests according to their target DNs. The proxy backend matches target DNs to base DNs that you specify in the configuration. If you specify multiple shards for data distribution, the proxy also forwards requests to the appropriate shard.

When specifying base DNs, bear in mind the following points:

In deployments where you must distribute data for horizontal write scalability, you can configure multiple service discovery mechanisms for the same base DNs. Each service discovery mechanism targets its own distinct shard of directory data.

To enable write scalability beyond what is possible with a single shard of replicas, each shard must have its own independent replication configuration. Replication replays each update only within the shard.

The proxy backend distributes write requests across the shard, and balances the load within each shard:

Data distribution for horizontal scalability has the following properties:

A directory proxy server balances the LDAP request load across the remote LDAP servers.

The proxy performs load balancing for the following purposes:

The proxy applies these load balancing capabilities in hierarchical order to route requests. Follow the paths of requests through the proxy backend to a directory server in this diagram:

Notice in the diagram how the load balancing capabilities work together to route a request to a directory server:

Proxy services are designed to proxy user data rather than server configuration or monitoring data. A proxy server must expose the same LDAP schema for user data as the remote directory servers.

If the schema definitions for user data differ between the proxy server and the remote directory servers, you must update them to bring them into alignment.

For details on DS server schema, refer to LDAP schema.

If the remote servers are not DS servers, refer to the schema documentation for the remote servers. Schema formats are not identical across all LDAPv3 servers. You will need to translate the differences into native formats, and apply changes locally on each server.

Create proxy backends only on servers set up as proxy servers (with setup --profile ds-proxy-server).

There are many specific ways that an LDAP request can fail. Not all failure result codes indicate a permanent failure, however. The following result codes from a remote directory server indicate a temporary failure:

When a forwarded request finishes with one of these return codes, the proxy backend retries the request on another server. In this case, the client does not receive the result from the remote directory server.

When a forwarded request finishes with a permanent server-side error return code, the proxy backend returns the result to the client application. In this case, the client must handle the error.

Connection failures can prevent the remote directory server from responding at all. The proxy backend handles connection failures differently, depending on whether it is inherently safe to replay the operation:

A proxy backend protects against failed and stale connections with feedback from periodic availability-check and keep-alive requests to remote directory servers.

The requests serve these purposes:

If necessary, you can configure how often such requests are sent using the proxy backend configuration properties.

When you deploy a directory proxy server, you limit the mechanisms for access control and resource limits. Such mechanisms cannot rely on ACIs in user data, resource limit settings on user entries, or group membership to determine what the server should allow. They can depend only on the server configuration and on the properties of the incoming request.

Global access control policies provide a mechanism suitable for directory proxy servers. For details, refer to Access control.

Resource limits set in the server configuration provide a way to prevent directory clients from using an unfair share of system resources. For details on how to update a server’s configuration, refer to Resource limits.

Directory services are designed for basic availability. Directory data replication makes it possible to read and write directory data when the network is partitioned, where remote servers cannot effectively contact each other. This sort of availability assumes that client applications can handle temporary interruptions.

Some client applications can be configured to connect to a set of directory servers. Some of these clients are able to retry requests when a server is unavailable. Others expect a single point of entry to the directory service, or cannot continue promptly when a directory server is unavailable.

Individual directory servers can become unavailable for various reasons:

All of these interruptions must be managed on the client side. Some could require reconfiguration of each client application.

A directory proxy server provides high availability to client applications by hiding these implementation details from client applications. The proxy presents client applications with a uniform view of the remote directory servers. It periodically rereads the remote servers' configurations and checks connections to route requests correctly despite changes in server configurations and availability.

Directory proxy servers are well-suited for applications that need a single entry point to the directory service.

Unlike directory servers with their potentially large sets of volatile user data, directory proxy servers manage only their configuration state. Proxy servers can start faster and require less disk and memory space. Each directory proxy server can effectively be a clone of every other, with a configuration that does not change after server startup. Each clone routes LDAP requests and handles problems in the same way.

When you configure all directory proxy servers as clones of each other, you have a number of identical directory access points. Each point provides the same view of the underlying directory service. The points differ from each other only by their connection coordinates (host, port, and potentially key material to establish secure connections).

To consolidate identical access points to a single access point, configure your network to use a virtual IP address for the proxy servers. You can restart or replace proxy servers at any time, in the worst case losing only the client connections that were established with the individual proxy server.

An alternative to a single, global access point for all applications is to repeat the same approach for each key application. The proxy server configuration is specific to each key application. Clones with that configuration provide a single directory access point for the key application. Other clones do the same other for other key applications. Be sure to provide a more generic access point for additional applications.

Data distribution through the proxy comes with the following constraints:

The example that follows is intended for evaluation on a single computer. It involves installing five DS servers and one AM server:

As you follow the example, notice the following characteristics:

DS servers have authorization features that rely on information about the client application connection, such as the client IP address and the security strength factor (SSF). DS ACIs, and lists of allowed, restricted, and denied clients use this information, for example.

When running DS behind a software load balancer, such as HAProxy, or AWS Elastic Load Balancing, the load balancer does network address translation. The load balancer connects to the DS server, and the DS does not have access to the client application connection. Cloud deployments often use a load balancer in this way.

Software load balancers that implement the proxy protocol can transfer the original client connection information through the protocol to the DS server. If your deployment uses a software load balancer and any features that rely on client connection information, enable the protocol in the load balancer, and configure proxy protocol support in DS.

By default, support for the proxy protocol is disabled. You must enable and configure the feature.

Once you set proxy-protocol-enabled:true, make sure you include all the load balancers in the list of proxy-protocol-allowed-client values. DS or the connection handler you configured accepts only connections from these allowed clients, and only if they use the proxy protocol.

DS servers rely on data replication for high availability with tolerance for network partitions. The directory service continues to allow both read and write operations when the network is down. As a trade off, replication provides eventual consistency, not immediate consistency.

A load balancer configured to distribute connections or requests equitably across multiple servers can therefore cause an application to get an inconsistent view of the directory data. This problem arises in particular when a client application uses a pool of connections to access a directory service:

The following sequence diagram illustrates the race condition:

When used in failover mode, also known as active/passive mode, this problem is prevented. However, the load balancer adds network latency while reducing the number of directory servers actively used. This is unfortunate, since the directory server replicas are designed to work together as a pool.

Unlike many load balancers, ForgeRock Identity Platform software has the capability to account for this situation, and to balance the request load appropriately across multiple directory servers.

Apply the following recommendations in your directory service deployments: