Create connector configurations using REST with the createCoreConfig and createFullConfig actions. For more information, see Configure Connectors Using REST.

To set up a synchronization mapping using the IDM admin UI:

When you use the UI to map a property with an encrypted value, you are prompted to set up a transformation script to decrypt the value when that property is synchronized. The resulting mapping looks similar to the following, which shows the transformation of a user’s password property:

Scriptable conditions create mapping logic, based on the result of the condition script. If the script does not return true, IDM does not manipulate the target attribute during a synchronization operation.

In the following excerpt, the value of the target mail attribute is set to the value of the source email attribute only if the source attribute is not empty:

The most common use of synchronization scripts is when a target object is created or updated.

The onUpdate script is always called for an UPDATE situation, even if the synchronization process determines that there is no difference between the source and target objects, and that the target object will not be updated.

If the onUpdate script has run and the synchronization process determines that the target value to set is the same as its existing value, the change is prevented from synchronizing to the target.

The following excerpt of a sample mapping derives a DN for an LDAP entry when the corresponding managed entry is created:

Construct and Manipulate Attributes With Scripts shows how to manipulate attributes with scripts when objects are created and updated. You can also trigger scripts in response to other synchronization actions. For example, you might not want to delete a managed user directly when an external account is deleted, but instead unlink the objects and deactivate the user in another resource. Alternatively, you might delete the object in IDM and run a script to perform some subsequent action.

The following example shows a more advanced mapping configuration that exposes the script hooks available during synchronization:

The following list shows the properties that you can use as hooks in mapping configurations to call scripts:

Scripts can obtain data from any connected system by using the openidm.read(id) function, where id is the identifier of the object to read.

The following example reads a managed user object from the repository:

The following example reads an account from an external LDAP resource:

IDM provides a logger object that you can use from scripts defined in your mapping. These scripts can log messages to the OSGi console and to log files, which are then ingested by the monitoring/logs endpoint. The logger object includes the following functions:

Consider the following mapping excerpt:

The scripts that are defined for onCreate, onUpdate, and result log a warning message to the console whenever an object is created or updated, or when a result is returned. The script result includes the full source object.

The scripts that are defined in the properties section of the mapping log a warning message if the property in the source object is missing or empty. The last script logs a warning message that includes the description of the source object.

During a reconciliation operation, these scripts would generate output in the OSGi console, similar to the following:

In addition to the static synchronization actions described previously, you can provide a script to run in specific synchronization situations. You can specify the script inline (with the "source" property).

The following excerpt of a sample mapping specifies that an inline script should be invoked when a synchronization operation assesses an entry as ABSENT in the target system. The script checks whether the employeeType property of the corresponding source entry is contractor. If so, the source entry is ignored. Otherwise, the entry is created on the target system:

The following variables are available to a script that is called as an action:

For more information about the variables available to scripts, refer to Script variables.

The result obtained from evaluating this script must be a string whose value is one of the synchronization actions listed in Synchronization actions. This resulting action is shown in the reconciliation log.

To launch a script as a synchronization action using the IDM admin UI:

IDM processes a correlation query by constructing a query map. The content of the query is generated dynamically, using values from the source object. For each source object, a new query is sent to the target system, using (possibly transformed) values from the source object for its execution.

Queries are run against target resources, either managed or system objects, depending on the mapping. Correlation queries on system objects access the connector, which executes the query on the external resource.

You express a correlation query using a query filter (_queryFilter). For more information about query filters, refer to Define and call data queries. The synchronization process executes the correlation query to search through the target system for objects that match the current source object.

To configure a correlation query, define a script whose source returns a query that uses the _queryFilter, for example:

In general, a correlation query should meet the requirements of most deployments. However, if you need a more powerful correlation mechanism than a simple query can provide, you can write a correlation script with additional logic. Correlation scripts can be useful if your query needs extra processing, such as fuzzy-logic matching or out-of-band verification with a third-party service over REST. Correlation scripts are generally more complex than correlation queries, and impose no restrictions on the methods used to find matching objects.

A correlation script must execute a query and return the result of that query. The result of a correlation script is a list of maps, each of which contains a candidate _id value. If no match is found, the script returns a zero-length list. If exactly one match is found, the script returns a single-element list. If there are multiple ambiguous matches, the script returns a list with multiple elements. There is no assumption that the matching target record or records can be found by a simple query on the target system. All of the work required to find matching records is left to the script.

To invoke a correlation script, use one of the following properties:

The following example triggers a reconciliation operation over REST based on the systemLdapAccounts_managedUser mapping:

By default, a reconciliation run ID is returned immediately when the reconciliation operation is initiated. Clients can make subsequent calls to the reconciliation service, using this reconciliation run ID to query its state, and to call operations on it. For an example, refer to Reconciliation Details.

The reconciliation run initiated previously would return something similar to the following:

To complete the reconciliation operation before the reconciliation run ID is returned, set the waitForCompletion property to true when the reconciliation is initiated:

To cancel an in progress reconciliation, specify the reconciliation run ID. The following REST call cancels the reconciliation run initiated in the previous section:

The output for a reconciliation cancellation request is similar to the following:

If the reconciliation run is waiting for completion before its ID is returned, obtain the reconciliation run ID from the list of active reconciliations, as described in the following section.

Display a list of reconciliation processes that have completed, and those that are in progress, by running a RESTful GET on "https://<tenant-env-fqdn>/openidm/recon".

The following example displays all reconciliation runs:

You can adjust the number of reconciliation runs that are stored in IDM by adding the maxAnalysisRunsPerMapping and maxNonAnalysisRunsPerMapping properties to your mapping:

In this context, analysis refers to reconciliation runs that are triggered with the analyze=true parameter. These runs don’t perform any actions, but determine which actions would be performed in a real reconciliation. Non-analysis refers to a normal reconciliation.

In contrast, the IDM admin UI displays the results of only the most recent reconciliation. For more information, refer to idm-synchronization:manage-recon.adoc#recon-details-ui.

To display the details of a specific reconciliation over REST, include the reconciliation run ID in the URL. The following call shows the details of the reconciliation run initiated in idm-synchronization:manage-recon.adoc#triggering-recons.

LiveSync operations are specific to a system object type (such as system/ldap/account). Apart from scheduling liveSync, as described in Scheduling LiveSync Through the UI, you can launch a liveSync operation on demand for a particular system object type as follows:

To troubleshoot a liveSync operation that has not succeeded, include the detailedFailure parameter to return additional information. For example:

The first time liveSync is called, it does not have a synchronization token in the database to establish which changes have already been processed. The default liveSync behavior is to locate the last existing entry in the change log, and to store that entry in the database as the current starting position from which changes should be applied. This behavior prevents liveSync from processing changes that might already have been processed during an initial data load. Subsequent liveSync operations will pick up and process any new changes.

Typically, in setting up liveSync on a new system, you would load the data initially (by using reconciliation, for example) and then enable liveSync, starting from that base point.

In the case of DS, the change log (cn=changelog) can be read only by uid=admin by default. If you are configuring liveSync with DS, the principal that is defined in the LDAP connector configuration must have access to the change log. For information about allowing a regular user to read the change log, refer to Allow a User or Application to Read the Change Log.

If the following error message is present, you might have forgotten to set changelog-read access for a regular user:

You can filter the source or target objects that are included in a synchronization operation using the validSource, validTarget, or sourceCondition properties in your mapping:

During synchronization, scripts and filters have access to a source object and a target object. Examples already shown in this section use source.attributeName to retrieve attributes from the source objects. Scripts can also write to target attributes using target.attributeName syntax, for example:

The sourceCondition filter also has the linkQualifier variable in its scope.

For more information about functions in identity-related scripts, refer to scripting functions.

Every reconciliation operation performs a query on the source and on the target resource, to determine which records should be reconciled. The default source and target queries are _queryFilter=true&_fields=_id, which means that all records in both the source and the target are considered candidates for that reconciliation operation.

You can restrict reconciliation to specific entries by defining an explicit sourceQuery or targetQuery in the mapping configuration.

For example, to restrict reconciliation to those records whose employeeType on the source resource is Permanent, you might specify a source query as follows:

The format of the query can be any query type that is supported by the resource, and can include additional parameters, if applicable. Use the _queryFilter parameter, in common filter notation.

The source and target queries send the query to the resource that is defined for that source or target, by default. You can override the resource the query is sent to by specifying a resourceName in the query. For example, to query a specific endpoint instead of the source resource, you might modify the preceding source query as follows:

To override a source or target query that is defined in the mapping, you can specify the query when you call the reconciliation operation. For example, to reconcile all employee entries, and not just the permanent employees, you would run the reconciliation operation as follows:

By default, a reconciliation operation runs both the source and target phase. To avoid queries on the target resource, set runTargetPhase to false in the mapping configuration. To prevent the target resource from being queried during the reconciliation operation configured in the previous example, amend the mapping configuration as follows:

You can restrict reconciliation to a specific record in much the same way as you restrict reconciliation by using queries.

To restrict reconciliation to a specific ID, use the reconById action, instead of the recon action when you call the reconciliation operation. Specify the ID with the id parameter.

The following command reconciles only the user with ID b3c2f414-e7b3-46aa-8ce6-f4ab1e89288c, for the mapping managedUser_systemLdapAccounts. The command synchronizes this particular user account in LDAP with the data from the managed user repository. The example assumes that implicit synchronization has been disabled, and that a reconciliation operation is required to copy changes made in the repository to the LDAP system:

Reconciliation by ID takes the default reconciliation options that are specified in the mapping, so the source and target queries, and source and target phases apply equally to reconciliation by ID.

For a mapping that has managed objects as the source, an implicit synchronization is triggered if any source property changes, regardless of whether the modified property is explicitly defined as a source property in the mapping.

This default behavior is helpful in situations where no source properties are explicitly defined—any property within the object is included as part of the mapping.

However, this behavior adds a processing overhead, because every mapping from the managed object is invoked when any managed object property changes. If several mappings are configured from the managed object, this default behavior can cause performance issues.

In these situations, you can restrict the properties that should trigger an implicit synchronization per mapping, using the triggerSyncProperties attribute. This attribute contains an array of JSON pointers to the properties that must change before an implicit synchronization to the target is triggered. If none of these properties changes, no synchronization is triggered, even if other properties in the object change.

In the following example, implicit synchronization is triggered only if the mail, telephoneNumber, or userName of an object changes:

If any other property changes on the managed object, no implicit synchronization is triggered.

By default, all mappings are automatically synchronized. A change to a managed object is automatically synchronized to all resources for which the managed object is configured as a source. If liveSync is enabled for a system, changes to an object on that system are automatically propagated to the managed object repository.

To prevent automatic synchronization for a specific mapping, set the enableSync property of that mapping to false. In the following example, implicit synchronization is disabled. This means that changes to objects in the internal repository are not automatically propagated to the LDAP directory. To propagate changes to the LDAP directory, reconciliation must be launched manually:

If enableSync is set to false for a mapping from a system resource to managed/realm-name_user (for example "systemLdapAccounts_managedUser"), liveSync is disabled for that mapping.

If a liveSync operation fails, IDM reattempts the change an infinite number of times until the change is successful. This behavior can increase data consistency in the case of transient failures (for example, when the connection to the database is temporarily lost). However, in situations where the cause of the failure is permanent (for example, if the change does not meet certain policy requirements) the change will never succeed, regardless of the number of attempts. In this case, the infinite retry behavior can effectively block subsequent liveSync operations from starting.

To avoid this, you can configure a liveSync retry policy to specify the number of times a failed modification should be reattempted, and what should happen if the modification is unsuccessful after the specified number of attempts.

Ultimately, a scheduled reconciliation operation forces consistency; however, to prevent repeated retries that block liveSync, restrict the number of times you attempt the same modification. You can then specify what happens to failed liveSync changes.

The failed modification can be stored in a dead letter queue , discarded, or reapplied. Alternatively, an administrator can be notified of the failure by email or by some other means. This behavior can be scripted. The default configuration in the samples provided with IDM is to retry a failed modification five times, and then to log and ignore the failure.

You configure the liveSync retry policy in the connector configuration. The sample connector configurations have a retry policy defined as follows:

The following liveSync retry policy configuration specifies a maximum of four retries before the failed modification is sent to the dead letter queue:

In the case of a failed modification, a message similar to the following is output to the logs:

IDM reattempts the modification the specified number of times. If the modification is still unsuccessful, a message similar to the following is logged:

The log message indicates the entry for which the modification failed (uid=jdoe, in this example).

You can view the failed modification in the dead letter queue, over the REST interface, as follows:

To view the details of a specific failed modification, include its ID in the URL:

By default, IDM implicitly synchronizes managed object changes out to all resources for which the managed object is configured as a source. If there are several targets that must be synchronized, these targets are synchronized one at a time, one after the other. If any of the targets is remote or has a high latency, the implicit synchronization operations can take some time, delaying the successful return of the managed object change.

To decouple the managed object changes from the corresponding synchronizations, you can configure queued synchronization, which persists implicit synchronization events to the IDM repository. Queued events are then read from the repository and executed according to the queued synchronization configuration.

Because synchronization operations are performed in parallel, queued synchronization can improve performance if you have several fast, reliable targets. However, queued synchronization is also useful when your targets are slow or unreliable, because the managed object changes can complete before all targets have been synchronized.

The following illustration shows how synchronization operations are added to a local, in-memory queue. Note that this queue is distinct from the repository queue for synchronization events:

If implicit synchronization fails for a target resource (for example, due to a policy validation failure on the target or the target being unavailable), the synchronization operation stops at that point. In this scenario, a record might be changed in the repository and in the targets on which synchronization was successful but not on the failed target or on any targets that would have been synchronized after the failure. This can result in disparate data sets across resources. Although a reconciliation operation would bring all targets back in sync, reconciliation can be expensive with large data sets.

You can configure synchronization failure compensation to:

With this script:

Each scheduled reconciliation and liveSync task requires a schedule configuration, with the following format:

These properties are specific to the scheduler service, and are explained in Schedule tasks and events.

To schedule a reconciliation or liveSync task, set the invokeService property to either sync (for reconciliation) or provisioner for liveSync.

The value of the invokeContext property depends on the type of scheduled event. For reconciliation, the properties are set as follows:

The mapping is referenced by its name in the mapping configuration.

For liveSync, the properties are set as follows:

The source property follows the convention for a pointer to an external resource object, and takes the form system/resource-name/object-type.

To configure liveSync using the IDM admin UI, set up a liveSync schedule:

To specify that the reconciliation for a specific mapping should be distributed across a cluster, add the clusteredSourceReconEnabled property to the mapping and set it to true. For example:

By default, the number of records per page is 10000. You can also enable target query paging with the reconTargetQueryPaging property (defaults to false). To change the query page sizes, set the reconSourceQueryPageSize and reconTargetQueryPageSize properties.

The following example enables target query paging and changes the target and source query page sizes:

To set these properties using the IDM admin UI, click Configure > Mappings, select the mapping to change, and click the Advanced tab.

Clustered reconciliation has the following limitations:

The sourceProcessedByNode property indicates how many records are processed by each node. You can verify the load distribution per node by running a GET on the recon endpoint, for example:

You can also display the nodes responsible for each source page in the IDM admin UI. Click on the relevant mapping and expand the In Progress or Reconciliation Results item. The following image shows a clustered reconciliation in progress. The details include the number of records that have been processed, the current duration of the reconciliation, and the load distribution, per node:

You cancel a clustered reconciliation in the same way as a non-clustered reconciliation, for example:

When the cancellation has completed, a query on that reconciliation ID will show the state and stage of the reconciliation as follows:

In a clustered environment, all reconciliation operations are considered to be "cluster-friendly". This means that even if a mapping is configured as "clusteredSourceReconEnabled" : false, you can view the in progress operation on any node in the cluster, even if that node is not currently processing the reconciliation. You can also cancel a reconciliation in progress from any node in the cluster.

To optimize performance, reconciliation does not correlate source objects to target objects if the set of target objects is empty when the correlation is started. This considerably speeds up the process the first time reconciliation is run. You can change this behavior for a specific mapping by adding the correlateEmptyTargetSet property to the mapping definition and setting it to true. For example:

Be aware that this setting will have a performance impact on the reconciliation process.

All links are queried at the start of reconciliation and the results of that query are used. You can disable the link prefetching so that the reconciliation process looks up each link in the database as it processes each source or target object. To disable link prefetching, add the prefetchLinks property to your mapping, and set it to false:

Be aware that this setting will have a performance impact on the reconciliation process.

By default, reconciliation is multithreaded; numerous threads are dedicated to the same reconciliation run. Multithreading generally improves reconciliation performance. The default number of threads for a single reconciliation run is 10 (plus the main reconciliation thread). Under normal circumstances, you should not need to change this number. However the default might not be appropriate in the following situations:

To change the number of threads, set the taskThreads property in your mapping:

A zero value runs reconciliation as a serialized process, on the main reconciliation thread.

Reconciliation operations are processed in two phases; a source phase and a target phase. In most reconciliation configurations, source and target queries make a read call to every record on the source and target systems to determine candidates for reconciliation. On slow source or target systems, these frequent calls can incur a substantial performance cost.

To improve query performance in these situations, you can preload the entire result set into memory on the source or target system, or on both systems. Subsequent read queries on known IDs are made against the data in memory, rather than the data on the remote system. For this optimization to be effective, the entire result set must fit into the available memory on the system for which it is enabled.

The optimization works by defining a sourceQuery or targetQuery in the synchronization mapping that returns not just the ID, but the complete object.

The following example query loads the full result set into memory during the source phase of the reconciliation. The example uses a common filter expression, called with the _queryFilter keyword. The query returns the complete object:

IDM attempts to detect what data has been returned. The autodetection mechanism assumes that a result set that includes three or more fields per object (apart from the _id and rev fields) contains the complete object.

You can explicitly state whether a query is configured to return complete objects by setting the value of sourceQueryFullEntry or targetQueryFullEntry in the mapping. The setting of these properties overrides the autodetection mechanism.

Setting these properties to false indicates that the returned object is not the complete object. This might be required if a query returns more than three fields of an object, but not the complete object. Without this setting, the autodetect logic would assume that the complete object was being returned. IDM uses only the IDs from this query result. If the complete object is required, the object is queried on demand.

Setting these properties to true indicates that the complete object is returned. This setting is typically required only for very small objects, for which the number of returned fields does not reach the threshold required for the auto-detection mechanism to assume that it is a full object. In this case, the query result includes all the details required to pre-load the full object.

The following excerpt indicates that the full objects are returned and that IDM should not autodetect the result set:

By default, all the attributes defined in the connector configuration are loaded into memory. If your mapping uses only a small subset of the attributes in the connector configuration, you can restrict your query to return only those attributes required for synchronization by using the _fields parameter with the query filter.

The following excerpt loads only a subset of attributes into memory, for all users in an LDAP directory.

Improve Reconciliation Query Performance describes how to improve reconciliation performance by loading all entries into memory to avoid making individual requests to the external system for every ID. However, this optimization depends on the entire result set fitting into the available memory on the system for which it is enabled. For particularly large data sets (for example, data sets of hundreds of millions of users), having the entire data set in memory might not be feasible.

To alleviate this constraint, you can use reconciliation paging, which breaks down extremely large data sets into chunks. It also lets you specify the number of entries that should be reconciled in each chunk or page.

Reconciliation paging is disabled by default, and can be enabled per mapping. To configure reconciliation paging, set the reconSourceQueryPaging property to true and set the reconSourceQueryPageSize in the synchronization mapping:

The value of reconSourceQueryPageSize must be a positive integer, and specifies the number of entries that will be processed in each page. If reconciliation paging is enabled but no page size is set, a default page size of 1000 is used.

An object-mapping object specifies the configuration for a mapping of source objects to target objects. The name, source, and target properties are mandatory. Other properties are optional or implicit. They have a default value if not set.

By default, links are maintained between source and target objects in mappings. This behavior is governed by the enableLinking property in the mapping. If enableLinking is set to false, links are not maintained.

You may want to disable linking in the case of a bulk import.

To maintain links between source and target objects in mappings, IDM stores an object set in the repository. The object set identifier follows this scheme:

Here, mapping represents the name of the mapping for which links are managed.

Link entries have the following structure:

IDM performs the following queries on a link object set:

IDM performs reconciliation on a per-mapping basis. The process of reconciliation for a given mapping includes these stages:

External synchronized objects expose an API to request immediate synchronization. This API includes the following requests and responses.

Obtaining the Details of a Reconciliation describes how to obtain the details of a reconciliation run over REST. This section provides more information on the metrics returned when you query the recon endpoint. Reconciliation is processed as a series of distinct tasks. The durationSummary property indicates the period of time spent on each task. You can use this information to address reconciliation performance bottlenecks.

The following sample output shows the kind of information returned for each reconciliation run:

The specific reconciliation tasks that are run depend on the configuration for that mapping. For example, the sourcePagePhase is run only if paging is enabled. The linkQuery is run only for non-clustered reconciliation operations, because an initial query of all links does not make sense if a single source page query is being run.