--- title: Synchronization reference description: The synchronization engine is one of the core PingIDM services. You configure the synchronization service through a mappings property that specifies mappings between objects that are managed by the synchronization engine. component: pingoneaic page_id: pingoneaic:idm-synchronization:synchronization-ref canonical_url: https://docs.pingidentity.com/pingoneaic/idm-synchronization/synchronization-ref.html keywords: ["Synchronization", "Scripts"] section_ids: sync-object-mapping: Object-mapping objects mapping-object-properties: Mapping Object Properties sync-property-objects: Property Objects sync-policy-objects: Policy Objects sync-script-objects: Script Object sync-links: Links sync-queries: Queries sync-reconciliation-stages: Reconciliation Stages sync-rest-api: REST API reconciliation-metrics: Reconciliation Duration Metrics recon-tasks-ref: Recon Tasks recon-metrics-ref: Metrics Collected --- # Synchronization reference The synchronization engine is one of the core PingIDM services. You configure the synchronization service through a `mappings` property that specifies mappings between objects that are managed by the synchronization engine. ```json { "mappings": [ object-mapping object, ... ] } ``` ## Object-mapping objects An object-mapping object specifies the configuration for a mapping of source objects to target objects. The `name`, `source`, and `target` properties are mandatory. Other properties are optional or implicit. They have a default value if not set. ```json { "correlationQuery" : script object, "correlationScript" : script object, "displayName" : string, "enableLinking" : boolean, "enableSync" : boolean, "linkQualifiers" : [ list of strings ] or script object, "links" : string, "name" : string, "onCreate" : script object, "onDelete" : script object, "onLink" : script object, "onMapping" : script object, "onUnlink" : script object, "onUpdate" : script object, "optimizeAssignmentSync" : boolean, "policies" : [ policy object, ... ], "postMapping" : script object, "properties" : [ property object, ... ], "queuedSync" : { property object }, "reconProgressStateUpdateInterval" : integer, "reconSourceQueryPageSize" : integer, "reconSourceQueryPaging" : boolean, "reconTargetQueryPageSize" : integer, "reconTargetQueryPaging" : boolean, "result" : script object, "runTargetPhase" : boolean, "source" : string, "sourceCondition" : script object or queryFilter string, "sourceIdsCaseSensitive" : boolean, "sourceQuery" : [ list of properties ], "sourceQueryFullEntry" : boolean, "syncAfter" : [ list of strings ], "target" : string, "targetIdsCaseSensitive" : boolean, "targetQuery" : [ list of properties ], "targetQueryFullEntry" : boolean, "taskThreads" : integer, "triggerSyncProperties" : [ list of JSON pointers ], "validSource" : script object, "validTarget" : script object } ``` ### Mapping Object Properties * correlationQuery script object, optional A script that yields a query object to query the target object set when a source object has no linked target. The syntax for writing the query depends on the target system of the correlation. Learn more in [Writing Correlation Queries](chap-correlation.html#correlation-queries-configuring). The source object is provided in the `source` property in the script scope. * correlationScript script object, optional A script that goes beyond a `correlationQuery` of a target system. Used when you need another method to determine which records in the target system relate to the given source record. The syntax depends on the target of the correlation. Learn more about defining correlation scripts in [Writing Correlation Scripts](chap-correlation.html#correlation-scripts). * displayName string, optional The mapping name displayed in the UI. * enableLinking boolean, true or false Specifies whether [links](#sync-links) should be maintained between source and target objects for a mapping. Default : `true` * enableSync boolean, true or false Specifies whether automatic synchronization, such as liveSync and implicit synchronization, should be enabled for a specific mapping. Learn more in [Disable Automatic Synchronization Operations](chap-implicit-live-sync.html#disabling-automatic-sync). Default : `true` * linkQualifiers list of strings or script object, optional Enables mapping of a single source object to multiple target objects. Example: `"linkQualifiers" : [ "employee", "customer" ]` or ```json "linkQualifiers" : { "type" : "text/javascript", "globals" : { }, "source" : "if (returnAll) { ['contractor', 'employee', 'customer', 'manager'] } else { if(object.type === 'employee') { ['employee', 'customer', 'manager'] } else { ['contractor', 'customer'] } }" } ``` If a script object, the script must return a list of strings. * links string, optional Enables reuse of the links created in another mapping. Example: `"systemLdapAccounts_managedUser"` reuses the links created by a previous mapping whose `name` is `"systemLdapAccounts_managedUser"`. * name string, required Uniquely names the object mapping. Used in the link object identifier. * onCreate script object, optional A script to run when a target object is to be created after property mappings have been applied. In the root scope, the source object is provided in the `source` property, the projected target object in the `target` property, and the link situation that led to the create operation in the `situation` property. Properties on the target object can be modified by the script. If a property value isn't set by the script, IDM falls back on the default property mapping configuration. If the script throws an exception, the target object creation is aborted. * onDelete script object, optional A script to run when a target object is to be deleted, after property mappings have been applied. In the root scope, the source object is provided in the `source` property, the target object in the `target` property, and the link situation that led to the delete operation in the `situation` property. If the script throws an exception, the target object deletion is aborted. * onLink script object, optional A script to run when a source object is to be linked to a target object, after property mappings have been applied. In the root scope, the source object is provided in the `source` property and the projected target object in the `target` property. | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Although an `onLink` script has access to a copy of the target object, changes made to that copy won't be saved to the target system automatically. To persist changes made to target objects by an `onLink` script, you must explicitly include a call to the action that should be taken on the target object. For example, `openidm.create`, `openidm.update` or `openidm.delete`) within the script. | In the following example, when an LDAP target object is linked, the `"description"` attribute of that object is updated with the value `"Active Account"`. A call to `openidm.update` is made within the `onLink` script to set the value. ```json "onLink" : { "type" : "text/javascript", "source" : "target.description = 'Active Account'; openidm.update('system/ldap/account/' + target._id, null, target);" } ``` If the script throws an exception, target object linking is aborted. * onMapping script object, optional A script that is run as a part of a mapping. For example, a script that applies effective assignments as a part of the mapping. This is run after the mappings have been applied, but before `onUpdate` scripts are run. * onUnlink script object, optional A script to run when a source and a target object are to be unlinked after property mappings have been applied. In the root scope, the source object is provided in the `source` property and the target object in the `target` property. | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Although an `onUnlink` script has access to a copy of the target object, changes made to that copy won't be saved to the target system automatically. To persist changes made to target objects by an `onUnlink` script, you must explicitly include a call to the action that should be taken on the target object. For example, `openidm.create`, `openidm.update` or `openidm.delete` within the script. | In the following example, when an LDAP target object is unlinked, the `description` attribute of that object is updated with the value `Inactive Account`. A call to `openidm.update` is made within the `onUnlink` script to set the value. ```json "onUnlink" : { "type" : "text/javascript", "source" : "target.description = 'Inactive Account'; openidm.update('system/ldap/account/' + target._id, null, target);" } ``` If the script throws an exception, target object unlinking is aborted. * onUpdate script object, optional A script to run when a target object is to be updated after property mappings have been applied. In the root scope, the source object is provided in the `source` property, the projected target object in the `target` property, and the link situation that led to the update operation in the `situation` property. Any changes that the script makes to the target object will be persisted when the object is finally saved to the target resource. If the script throws an exception, the target object update is aborted. * optimizeAssignmentSync boolean When set to `true`, a synchronization event is triggered only by modifications to assignments if those modifications are relevant to that particular mapping. When set to `false`, the default, a synchronization event is triggered for all members of a role or assignment if the relationships, or attributes for an assignment are modified. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------- | | | Don't include `effectiveAssignments` in `triggerSyncProperties`, which negates all performance improvements provided by `optimizeAssignmentSync`. | * policies array of policy objects, optional Specifies a set of link conditions and associated actions to take in response. * postMapping script object, optional A script to run when sync has been performed on a managed object. This is run after the source object has been successfully synchronized with the target system. * properties array of property-mapping objects, optional Specifies mappings between source object properties and target object properties with optional transformation scripts. Learn more in [Property Object Properties](#sync-property-object-properties). * queuedSync list of properties, optional Specifies the [queued synchronization](chap-implicit-live-sync.html#queued-sync) configuration. * reconProgressStateUpdateInterval integer, optional Overrides the number of reconciliation operations required before the reconciliation progress state statistics are persisted to the repository. A value of `50` will write statistics to the repository for every `50` operations. Default : `1024`, minimum : `1` * reconSourceQueryPageSize integer Sets the page size for reconciliation source queries if paging is enabled. Default for clustered reconciliation : `10000` * reconSourceQueryPaging boolean, true or false Specifies whether paging should be used for reconciliation source queries. * Default for non-clustered reconciliation : `false` * Default for clustered reconciliation : `true` * reconTargetQueryPageSize integer Sets the page size for reconciliation target queries if paging is enabled. Default : `10000` * reconTargetQueryPaging boolean, true or false Specifies whether paging should be used for reconciliation target queries. * Default for non-clustered reconciliation : `false` * Default for clustered reconciliation : `true` * result script object, optional A script run after a reconciliation finishes. The variables available to a `result` script include: | Variable | Description | | --------------- | ------------------------------------------------------------------------------------------ | | `source` | Provides statistics about the source phase of the reconciliation. | | `target` | Provides statistics about the target phase of the reconciliation. | | `global` | Provides statistics about the entire reconciliation operation. | | `context` | Information related to the current operation, such as source and target. | | `mappingConfig` | A configuration object representing the mapping being processed. | | `reconState` | Provides the state of reconciliation operation, such as *success*, *failure*, or *active*. | * runTargetPhase boolean, true or false Specifies whether reconciliation operations should run both the source and target phase. To avoid queries on the target resource, set to `false`. Default : `true` * source string, required Specifies the path of the source objects. Example: `"managed/realm-name_user"`. * sourceCondition script object or `queryFilter` string, optional A script or `queryFilter` that determines if a source object should be included in the mapping. If no `sourceCondition` element or `validSource` script is specified, all source objects are included in the mapping. * sourceIdsCaseSensitive boolean, true or false Consider case sensitivity when linking source IDs. Only effective if this mapping defines links and ignored if the mapping reuses links to another mapping. Default : `true` * sourceQuery list of properties, optional Specifies the `queryFilter` to be used during reconciliation when querying the source objects. Learn more in [Restrict reconciliation by using queries](chap-restricting-sync.html#recon-by-query). * sourceQueryFullEntry boolean, true or false, optional Specifies whether the defined source query returns full object data (`true`) or IDs only (`false`). No default. If not set in the configuration, IDM will attempt to auto-detect the setting, based on the query results. * syncAfter list of strings, optional The specified mapping must be synchronized after all mappings in this list. * target string, required Specifies the path of the target objects. Example: `"system/ldap/account"`. * targetIdsCaseSensitive boolean, true or false Consider case sensitivity when linking target IDs. Only effective if this mapping defines links and ignored if the mapping reuses links. Default : `true` * targetQuery list of properties, optional Specifies the `queryFilter` to be used during reconciliation when querying the target objects. Learn more in [Restrict reconciliation by using queries](chap-restricting-sync.html#recon-by-query). * targetQueryFullEntry Boolean `true` or `false`, optional Specifies whether the defined target query returns full object data (`true`) or IDs only (`false`). No default. If not set in the configuration, IDM will attempt to auto-detect the setting based on the query results. * taskThreads integer, optional Sets the number of threads dedicated to the same reconciliation run. Default : `10` * triggerSyncProperties list, optional A list of `JsonPointers` to fields in the source object whose changes should trigger a synchronization operation. * validSource script object, optional A script that determines if a source object is valid to be mapped. The script yields a boolean value: `true` indicates the source object is valid and `false` can be used to defer mapping until some condition is met. In the root scope, the source object is provided in the `source` property. If the script isn't specified, then all source objects are considered valid. * validTarget script object, optional A script used during the target phase of reconciliation that determines if a target object is valid to be mapped. The script yields a boolean value: `true` indicates that the target object is valid and `false` indicates that the target object shouldn't be included in reconciliation. In the root scope, the target object is provided in the `target` property. If the script isn't specified, then all target objects are considered valid for mapping. ### Property Objects A property object specifies how the value of a target property is determined. ```json { "target" : string, "source" : string, "transform" : script object, "condition" : script object, "default": value } ``` Property Object Properties * target string, required Specifies the path of the property in the target object to map to. * source string, optional Specifies the path of the property in the source object to map from. If not specified, then the target property value is derived from the script or default value. * transform script object, optional A script to determine the target property value. The root scope contains the value of the source in the `source` property, if specified. If the `source` property has a value of `""`, the entire source object of the mapping is contained in the root scope. The resulting value yielded by the script is stored in the target property. * condition script object, optional A script to determine whether the mapping should be executed or not. The condition has an `"object"` property available in root scope, which (if specified) contains the full source object. For example `"source": "(object.email != null)"`. The script is considered to return a boolean value. * default any value, optional Specifies the value to assign to the target property if a non-null value isn't established by `source` or `transform`. If not specified, the default value is `null`. ### Policy Objects A policy object specifies a link condition and the associated actions to take in response. ```json { "condition" : optional, script object, "situation" : string, "action" : string or script object "postAction" : optional, script object } ``` Policy Object Properties * condition script object or queryFilter condition, optional Applies a policy, based on the link type, for example `"condition" : "/linkQualifier eq \"user\""`. A queryFilter condition can be expressed in two ways—as a string (`"condition" : "/linkQualifier eq \"user\""`) or a map, for example: ```json "condition" : { "type" : "queryFilter", "filter" : "/linkQualifier eq \"user\"" } ``` It is generally preferable to express a queryFilter condition as a map. A `condition` script has the following variables available in its scope: `object` and `linkQualifier`. * situation string, required Specifies the situation for which an associated action is to be defined. * action string or script object, required Specifies the action to perform. If a script is specified, the script is executed and is expected to yield a string containing the action to perform. The `action` script has the following variables available in its scope: `source`, `target`, `sourceAction`, `linkQualifier`, and `recon`. * postAction script object, optional Specifies the action to perform after the previously specified action has completed. The `postAction` script has the following variables available in its scope: `source`, `target`, `action`, `sourceAction`, `linkQualifier`, and `reconID`. `sourceAction` is `true` if the action was performed during the source reconciliation phase, and `false` if the action was performed during the target reconciliation phase. Learn more in [How Synchronization Situations Are Assessed](chap-situations-actions.html). | | | | - | ---------------------------------------------------------------------------------- | | | No `postAction` script is triggered if the `action` is either `IGNORE` or `ASYNC`. | #### Script Object Script objects take the following form. ```json { "type" : "text/javascript", "source": string } ``` * type string, required The script type. IDM supports `"text/javascript"`. * source string, required Specifies the source code of the script to be executed. ## Links By default, links are maintained between source and target objects in mappings. This behavior is governed by the `enableLinking` property in the mapping. If `enableLinking` is set to `false`, links are not maintained. You may want to disable linking in the case of a [bulk import](import-data.html). To maintain links between source and target objects in mappings, IDM stores an object set in the repository. The object set identifier follows this scheme: ``` links/mapping ``` Here, mapping represents the name of the mapping for which links are managed. Link entries have the following structure: ```json { "_id":string, "_rev":string, "linkType":string, "firstId":string "secondId":string, } ``` * \_id string The identifier of the link object. * \_rev string, required The value of link object's revision. * linkType string, required The type of the link. Usually the name of the mapping which created the link. * firstId string, required The identifier of the first of the two linked objects. * secondId string The identifier of the second of the two linked objects. ## Queries IDM performs the following queries on a link object set: 1. Find link(s) for a given firstId object identifier. ```sql SELECT * FROM links WHERE linkType = value AND firstId = value ``` Although a single result makes sense, this query is intended to allow multiple results so that this scenario can be handled as an exception. 2. Select link(s) for a given second object identifier. ```sql SELECT * FROM links WHERE linkType = value AND secondId = value ``` ## Reconciliation Stages IDM performs reconciliation on a per-mapping basis. The process of reconciliation for a given mapping includes these stages: 1. Iterate through all objects for the object set specified as `source`. For each source object, carry out the following steps. 1. Look for a link to a target object in the link object set, and perform a correlation query (if defined). 2. Determine the link condition, as well as whether a target object can be found. 3. Determine the action to perform based on the policy defined for the condition. 4. Perform the action. 5. Keep track of the target objects for which a condition and action has already been determined. 6. Write the results. 2. Iterate through all object identifiers for the object set specified as `target`. For each identifier, carry out the following steps: 1. Find the target in the link object set. Determine if the target object was handled in the first phase. 2. Determine the action to perform based on the policy defined for the condition. 3. Perform the action. 4. Write the results. 3. Iterate through all link objects, carrying out the following steps. 1. If the `reconId` is `"my"`, then skip the object. If the `reconId` is not recognized, then the source or the target is missing. 2. Determine the action to perform based on the policy. 3. Perform the action. 4. Store the `reconId` identifer in the mapping to indicate that it was processed in this run. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | To optimize a reconciliation operation, the reconciliation process does not attempt to correlate source objects to target objects if the set of target objects is empty when the correlation is started. For information on changing this default behaviour, refer to [Correlate Empty Target Sets](chap-performance.html#correlate-target-set). | ## REST API External synchronized objects expose an API to request immediate synchronization. This API includes the following requests and responses. * Request Example: ```http POST /openidm/system/csvfile/account/jsmith?_action=liveSync HTTP/1.1 ``` * Response (success) Example: ```http HTTP/1.1 204 No Content ... ``` * Response (synchronization failure) Example: ```http HTTP/1.1 409 Conflict ... [JSON representation of error] ``` ## Reconciliation Duration Metrics [Obtaining the Details of a Reconciliation](manage-recon.html#recon-details) describes how to obtain the details of a reconciliation run over REST. This section provides more information on the metrics returned when you query the `recon` endpoint. Reconciliation is processed as a series of distinct tasks. The `durationSummary` property indicates the period of time spent on each task. You can use this information to address reconciliation performance bottlenecks. The following sample output shows the kind of information returned for each reconciliation run: ```json { "_id": "3bc72717-a4bb-4871-b936-3a5a560c1a7c-37", "duration": 781561, "durationSummary": { "auditLog": { ... }, ... "sourceObjectQuery": { "count": 100, "max": 96, "mean": 14, "min": 6, "stdDev": 16, "sum": 1450 }, "sourcePagePhase": { "count": 1, "max": 20944, "mean": 20944, "min": 20944, "stdDev": 0, "sum": 20944 }, "sourceQuery": { "count": 1, "max": 120, "mean": 120, "min": 120, "stdDev": 0, "sum": 120 }, "targetPhase": { "count": 1, "max": 0, "mean": 0, "min": 0, "stdDev": 0, "sum": 0 }, "targetQuery": { "count": 1, "max": 19657, "mean": 19657, "min": 19657, "stdDev": 0, "sum": 19657 } }, ... } ``` The specific reconciliation tasks that are run depend on the configuration for that mapping. For example, the `sourcePagePhase` is run only if paging is enabled. The `linkQuery` is run only for non-clustered reconciliation operations, because an initial query of all links does not make sense if a single source page query is being run. ### Recon Tasks The following list describes all the possible tasks that can be run for a single reconciliation: * `sourcePhase` This phase runs only for non-clustered, non-paged reconciliations. The total duration (`sum`) is the time spent processing all records on the source system. * `sourcePagePhase` Queries and processes individual objects in a page, based on their IDs. This phase is run only for clustered reconciliations or for non-clustered reconciliations that have source paging configured. The total duration (`sum`) is the total time spent processing source pages across all cluster nodes. This processing occurs in parallel across all cluster nodes, so it is normal for the `sourcePagePhase` duration to exceed the total reconciliation duration. * `sourceQuery` Obtains all IDs on the source system, or in a specific source page. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | When the `sourceQuery` returns a null paging cookie, indicating that there are no more source IDs to reconcile, the clustered reconciliation process dispatches a scheduled job named `sourcePageCompletionCheck`.This job checks for remaining source page jobs on the scheduler. If there are no remaining source page jobs, the `sourcePageCompletionCheck` schedules the target phase. If there are still source page jobs to process, the `sourcePageCompletionCheck` schedules another instance of itself to perform these checks again after a few seconds.Because the target phase reconciles all IDs that were not reconciled during the source phase, it cannot start until all of the source pages are complete. Final reconciliation statistics cannot be generated and logged until all source page jobs have completed, so the `sourcePageCompletionCheck` runs even if the target phase is not enabled. | * `sourceObjectQuery` Queries the individual objects on the source system or page, based on their IDs. * `validSourceScript` Processes any scripts that should be run to determine if a source object is valid to be mapped. * `linkQuery` Queries any existing links between source and target objects. This phase includes the following tasks: * `sourceLinkQuery` Queries any existing links from source objects to target objects. * `targetLinkQuery` Queries any existing links from target objects that were not processed during the `sourceLinkQuery` phase. * `linkQualifiersScript` Runs any link qualifier scripts. For more information, refer to [Map a Single Source Object to Multiple Target Objects](linking-multiple-targets.html). * `onLinkScript` Processes any scripts that should be run when source and target objects are linked. * `onUnlinkScript` Processes any scripts that should be run when source and target objects are unlinked. * `deleteLinkObject` Deletes any links that are no longer relevant between source and target objects. * `correlationQuery` Processes any configured correlation queries. For more information, refer to [Writing Correlation Queries](chap-correlation.html#correlation-queries-configuring). * `correlationScript` Processes any configured correlation scripts. For more information, refer to [Writing Correlation Scripts](chap-correlation.html#correlation-scripts). * `onMappingScript` For roles, processes the script that applies the effective assignments as part of the mapping. * `activePolicyScript` Sets the action and active policy based on the current situation. * `activePolicyPostActionScript` Processes any scripts configured to run after policy validation. * `targetPhase` The aggregated result for time spent processing records on the target system. * `targetQuery` Queries all IDs on the target system. The list of IDs is restricted to IDs that have not already been linked to a source ID during the source phase. The target query generates a list of *orphan* IDs that must be reconciled if the target phase is not disabled. * `targetObjectQuery` Queries the individual objects on the target system, based on their IDs. * `validTargetScript` Processes any scripts that should be run to determine if a target object is valid to be mapped. * `onCreateScript` Processes any scripts that should be run when a new target object is created. * `updateTargetObject` Updates existing linked target objects, based on the configured situations and actions. * `onUpdateScript` Processes any scripts that should be run when a target object is updated. * `deleteTargetObject` Deletes any objects on the target resource that must be removed in accordance with the defined synchronization actions. * `onDeleteScript` Processes any scripts that should be run when a target object is deleted. * `resultScript` Processes the script that is executed when a reconciliation process has finished. * `propertyMappingScript` Runs any scripts configured for when source and target properties are mapped. * `postMappingScript` Processes any scripts that should be run when synchronization has been performed on the `managed/realm-name_user` object. * `onReconScript` Processes any scripts that should be run after source and target systems are reconciled. * `auditLog` Writes reconciliation results to the audit log. ### Metrics Collected For each phase, the following metrics are collected: * `count` The number of objects or records processed during that phase. For the `sourcePageQuery` phase, the `count` parameter refers to the page size. When the `count` statistic of a particular task refers to the number of records being reconciled, the `sum` statistic of that task represents the total time across the total number of threads running in all nodes in the cluster. For example: ```json "updateTargetObject": { "count": 1000000, "max": 1193, "mean": 35, "min": 11, "stdDev": 0, "sum": 35065991 } ``` * `max` The maximum time, in milliseconds, spent processing a record during that phase. * `mean` The average time, in milliseconds, spent processing a record during that phase. * `min` The minimum time, in milliseconds, spent processing a record during that phase. * `stdDev` The standard deviation, which measures the variance of the individual values from the mean. * `sum` The total amount of time, in milliseconds, spent during that phase.