Composite indexes

A PingDirectory server composite index can generate from multiple pieces of information, such as a combination of multiple filter components or a combination of filter components and a base distinguished name (DN), or from a single piece of information.

Because composite indexes are much more efficient than exploded indexes when dealing with large numbers of identical keys, use composite indexes for data sets where large numbers of entries (several thousand or more) share the same value for an indexed attribute.

To improve the performance of searches in directories with a large number of entries, especially with a large number of non-leaf entries, use equality composite indexes to combine a mandatory equality filter pattern with an optional base DN pattern.

Equality composite indexes offer two advantages over existing equality attribute indexes in the these types of deployments, a base DN pattern and index pages.

Base DN pattern

If a directory environment has many branches, but searches are often done within specific individual branches, use the base DN pattern to make search processing more efficient. The server only needs to search entries within a target branch.

For example, if the directory contains an "ou=Customers,dc=example,dc=com" branch with separate branches below that for sets of customers, such as "ou=ACME,ou=Customers,dc=example,dc=com" and "ou=SHOPCO,ou=Customers,dc=example,dc=com", you can define a composite index with a filter pattern of "(sn=?)" and a base DN pattern of "ou=?,ou=Customers,dc=example,dc=com". Then use a search with a filter of "(sn=Smith)" and a base DN of "ou=ACME,ou=Customers,dc=example,dc=com" to narrow the search to the Smiths in the ACME branch.

Index Pages

If many entries have the same value for a specific attribute, you can use composite indexes to break large ID sets up across multiple pages, unlike a traditional attribute index.

Using the previous example, if a search of the directory returns 50,000 Smiths, the results can be served in blocks of 5,000 IDs. An attribute index returns either:

One Smith record whose value is a block that contains all 50,000 of the matching entry IDs when the index isn’t exploded.

The exploded form is efficient for writing, for example, adding or removing a Smith involves just that one entry ID, but it is expensive for searching because it takes 50,000 reads to get all entry IDs for all of the Smiths.

50,000 Smith records that each have a value of the matching entry ID when the index is exploded.

The non-exploded form is efficient for searching because all of the entry IDs return in a single read, but it is expensive for writing. If a Smith must be added or removed, the entire block of 50,000 entry IDs must be rewritten.

Composite indexes break up the block of entry IDs across multiple pages, a page size of up to 5000. If the directory contains 50,000 Smiths, instead of having to choose between one block of 50,000 IDs or 50,000 blocks of one ID, the composite index returns 10 blocks of 5,000 IDs. This improves the efficiency of a read or write across many entries.

There is little performance overhead to the paging mechanism. Use an equality composite index for an attribute that has a lot of entries that have the same value, such as givenName or sn. Do not use a equality composite index for an attribute with very few entries with the same value, such as id or mail. For attributes in which all of the values match a small number of entries, use an equality attribute index.

Rebuilding indexes

When rebuilding specific non-attribute indexes, it is important to include any appropriate prefixes or postfixes with the index name to avoid errors.

To rebuild a composite index, use the rebuild-index tool. The index name must be preceded by composite.

bin/rebuild-index --index composite.thisIndexName

For the exact name to use in rebuilding the index, run the dbtest tool with the list-database-containers option and --backendID property.

bin/dbtest list-database-containers --backendID {backendID}

Define the following properties when configuring a composite index.

Composite Index Properties Description

Composite Index Properties	Description
`index-filter-pattern` (required)	Specifies a single-valued filter property used to identify a portion of the index criteria. You can only specify this at the time that the index definition is created.
`index-base-dn-pattern` (optional)	Specifies a single-valued DN property that indicates that the index is scoped to a specific subtree or subtree pattern. You can only specify this at the time the index definition is created.

index-filter-pattern (required)

Specifies a single-valued filter property used to identify a portion of the index criteria. You can only specify this at the time that the index definition is created.

index-base-dn-pattern (optional)

Specifies a single-valued DN property that indicates that the index is scoped to a specific subtree or subtree pattern. You can only specify this at the time the index definition is created.

Composite index filter patterns

You can use the following filter patterns to define a composite index.

Presence matching

Using a presence matching filter causes the server to index the specified attribute for presence matching, which identifies every entry that contains any value for the target attribute. The server uses this index to process searches with presence components.

Syntax and usage: Presence components use the following syntax: (attributeName=*). You can use a standalone presence component in a simple filter or as part of an AND filter pattern, for example "(&(objectClass=?)(attributeName=*))".

You can replace existing presence attribute indexes with composite indexes for improved scalability or to limit the scope of index keys by using a base DN pattern.

Equality matching

Using an equality matching filter causes the server to index all unique values for the specified attribute for equality matching. The server uses this index to process searches with:

Equality filter components that target the specified attribute
Greater-than-or-equal or less-than-or-equal components that target the specified attribute
Substring components with a subInitial (starts with) element that targets the specified attribute

Substring filter components can also have subAny (contains) and subFinal (ends with) elements, but the server requires a subInitial element to use a composite index with these types of filters.

Syntax and usage: Equality components use the following syntax: (attributeName=?). You can use an equality component in a simple filter or as part of an AND filter, either with multiple equality components or combined with other supported filter pattern components.

For AND filters that contain multiple equality components:

You can add an ordering or substring matching component, but you must place it last in the filter.
All other components in the filter must be equality matching components.

Learn more about when to use a composite equality index instead of an attribute equality index in Composite indexes.

Static equality matching

Using a static equality filter causes the server to index all entries that have the specified value for the given attribute. The server uses this index to process searches with that specific equality filter component.

Syntax and usage: Static equality components use the following syntax: (attributeName={staticValue}). You can use a static equality component in a simple filter or as part of an AND filter, either with multiple static equality components or combined with other supported filter pattern components.

Use static equality matching to index specific attribute values that are present in a large number of entries.

Substring matching

Using a substring matching filter causes the server to index values for the specified attribute for substring matching with filters that include subAny (contains) or subFinal (ends with) components. The server doesn’t use substring matching to index subInitial (starts with) components because they are better handled with equality filter patterns.

The server can use a composite index with this filter pattern to process any searches with substring filter components that target the specified attribute. For filters with subInitial (starts with) elements, the server might prefer to search using an available equality index for the attribute type, if one is available.

Syntax and usage: Substring components use the following syntax: (attributeName=*?*). You can only use one substring component in a given filter pattern, and it needs to be the last component of the pattern.

Approximate matching

Using an approximate matching filter causes the server to index all unique values for the specified attribute for approximate matching. The server uses this index to process searches with approximate matching filter components that target the specified attribute.

Syntax and usage: Approximate matching components use the following syntax: (attributeName~=?). You can use an approximate matching component in a simple filter or as part of an AND filter pattern, for example "(&(givenName~=?)(sn~=?))".

You can replace existing approximate matching attribute indexes with composite indexes for improved scalability or to limit the scope of index keys by using a base DN pattern.

Guidelines for using AND filter patterns

You can specify an AND filter pattern with one or more presence, wildcard equality, static equality, or wildcard approximate matching filters, followed by a maximum of one wildcard substring filter. Each filter component inside the AND must specify a different attribute type.

Using this filter pattern causes the server to index entries with at least one value for each specified attribute. The server uses this index to process searches using AND filters that contain all but the last of those attribute types in equality filters and that contain the last attribute type in an appropriate filter for the wildcard filter.

For example, the server will use a composite index with filter pattern "(&(tenantID=?)(sn=?))" to process AND filters containing a tenantID equality filter and either an sn equality filter, an sn ordering filter, or an sn substring filter with at least a subInitial (starts with) element.