Synchronization reference

The synchronization engine is one of the core IDM services. You configure the synchronization service through a mappings property that specifies mappings between objects that are managed by the synchronization engine.

{
  "mappings": [ object-mapping object, ... ]
}

Object-mapping objects

An object-mapping object specifies the configuration for a mapping of source objects to target objects. The name, source, and target properties are mandatory. Other properties are optional or implicit (that is, they have a default value if not set).

{
  "correlationQuery"                 : script object,
  "correlationScript"                : script object,
  "displayName"                      : string,
  "enableLinking"                    : boolean,
  "enableSync"                       : boolean,
  "linkQualifiers"                   : [ list of strings ] or script object,
  "links"                            : string,
  "name"                             : string,
  "onCreate"                         : script object,
  "onDelete"                         : script object,
  "onLink"                           : script object,
  "onMapping"                        : script object,
  "onUnlink"                         : script object,
  "onUpdate"                         : script object,
  "optimizeAssignmentSync"           : boolean,
  "policies"                         : [ policy object, ... ],
  "postMapping"                      : script object,
  "properties"                       : [ property object, ... ],
  "queuedSync"                       : { property object },
  "reconAssociation"                 : { property object },
  "reconProgressStateUpdateInterval" : integer,
  "reconSourceQueryPageSize"         : integer,
  "reconSourceQueryPaging"           : boolean,
  "reconTargetQueryPageSize"         : integer,
  "reconTargetQueryPaging"           : boolean,
  "result"                           : script object,
  "runTargetPhase"                   : boolean,
  "source"                           : string,
  "sourceCondition"                  : script object or queryFilter string,
  "sourceIdsCaseSensitive"           : boolean,
  "sourceQueryFullEntry"             : boolean,
  "syncAfter"                        : [ list of strings ],
  "target"                           : string,
  "targetIdsCaseSensitive"           : boolean,
  "targetQueryFullEntry"             : boolean,
  "taskThreads"                      : integer,
  "triggerSyncProperties"            : [ list of JSON pointers ],
  "validSource"                      : script object,
  "validTarget"                      : script object
}

Mapping object properties

correlationQuery

script object, optional

A script that yields a query object to query the target object set when a source object has no linked target. The syntax for writing the query depends on the target system of the correlation. For examples of correlation queries, refer to Writing Correlation Queries. The source object is provided in the source property in the script scope.

correlationScript

script object, optional

A script that goes beyond a correlationQuery of a target system. Used when you need another method to determine which records in the target system relate to the given source record. The syntax depends on the target of the correlation. For information about defining correlation scripts, refer to Writing Correlation Scripts.

displayName

string, optional

The mapping name displayed in the UI.

enableLinking

boolean, true or false

Specifies whether links should be maintained between source and target objects for a mapping.

Default : true

enableSync

boolean, true or false

Specifies whether automatic synchronization (liveSync and implicit synchronization) should be enabled for a specific mapping. For more information, refer to Disable Automatic Synchronization Operations.

Default : true

linkQualifiers

list of strings or script object, optional

Enables mapping of a single source object to multiple target objects.

Example: "linkQualifiers" : [ "employee", "customer" ] or

"linkQualifiers" : {
        "type" : "text/javascript",
        "globals" : { },
        "source" : "if (returnAll) {
                      ['contractor', 'employee', 'customer', 'manager']
                    } else {
                      if(object.type === 'employee') {
                        ['employee', 'customer', 'manager']
                      } else {
                        ['contractor', 'customer']
                      }
                    }"
      }

If a script object, the script must return a list of strings.

links

string, optional

Enables reuse of the links created in another mapping. Example: "systemLdapAccounts_managedUser" reuses the links created by a previous mapping whose name is "systemLdapAccounts_managedUser".

name

string, required

Uniquely names the object mapping. Used in the link object identifier.

onCreate

script object, optional

A script to execute when a target object is to be created, after property mappings have been applied. In the root scope, the source object is provided in the source property, the projected target object in the target property, and the link situation that led to the create operation in the situation property. Properties on the target object can be modified by the script. If a property value is not set by the script, IDM falls back on the default property mapping configuration. If the script throws an exception, the target object creation is aborted.

onDelete

script object, optional

A script to execute when a target object is to be deleted, after property mappings have been applied. In the root scope, the source object is provided in the source property, the target object in the target property, and the link situation that led to the delete operation in the situation property. If the script throws an exception, the target object deletion is aborted.

onLink

script object, optional

A script to execute when a source object is to be linked to a target object, after property mappings have been applied. In the root scope, the source object is provided in the source property, and the projected target object in the target property.

Note that, although an onLink script has access to a copy of the target object, changes made to that copy will not be saved to the target system automatically. If you want to persist changes made to target objects by an onLink script, you must explicitly include a call to the action that should be taken on the target object (for example openidm.create, openidm.update or openidm.delete) within the script.

In the following example, when an LDAP target object is linked, the "description" attribute of that object is updated with the value "Active Account". A call to openidm.update is made within the onLink script, to set the value.

"onLink" : {
    "type" : "text/javascript",
    "source" : "target.description = 'Active Account';
               openidm.update('system/ldap/account/' + target._id, null, target);"
}

If the script throws an exception, target object linking is aborted.

onMapping

script object, optional

A script that is run as part of a mapping. For example, a script that applies effective assignments as part of the mapping. This is run after the mappings have been applied, but before onUpdate scripts are executed.

onUnlink

script object, optional

A script to execute when a source and a target object are to be unlinked, after property mappings have been applied. In the root scope, the source object is provided in the source property, and the target object in the target property.

Although an onUnlink script has access to a copy of the target object, changes made to that copy will not be saved to the target system automatically. If you want to persist changes made to target objects by an onUnlink script, you must explicitly include a call to the action that should be taken on the target object (for example, openidm.create, openidm.update or openidm.delete) within the script.

In the following example, when an LDAP target object is unlinked, the description attribute of that object is updated with the value Inactive Account. A call to openidm.update is made within the onUnlink script, to set the value.

"onUnlink" : {
    "type" : "text/javascript",
    "source" : "target.description = 'Inactive Account';
               openidm.update('system/ldap/account/' + target._id, null, target);"
}

If the script throws an exception, target object unlinking is aborted.

onUpdate

script object, optional

A script to execute when a target object is to be updated, after property mappings have been applied. In the root scope, the source object is provided in the source property, the projected target object in the target property, and the link situation that led to the update operation in the situation property. Any changes that the script makes to the target object will be persisted when the object is finally saved to the target resource. If the script throws an exception, the target object update is aborted.

optimizeAssignmentSync

boolean

When set to true, a synchronization event is only triggered by modifications to assignments if those modifications are relevant to that particular mapping. When set to false (the default), a synchronization event is triggered for all members of a role or assignment if the relationships or attributes for an assignment are modified.

Don’t include effectiveAssignments in triggerSyncProperties as it negates all performance improvements provided by optimizeAssignmentSync.

policies

array of policy objects, optional

Specifies a set of link conditions and associated actions to take in response.

postMapping

script object, optional

A script to execute when sync has been performed on a managed object. This is run after the source object has been successfully synchronized with the target system.

properties

array of property-mapping objects, optional

Specifies mappings between source object properties and target object properties, with optional transformation scripts. refer to Property Object Properties.

queuedSync

list of properties, optional

Specifies the queued synchronization configuration.

reconAssociation

list of properties, optional

Specifies the recon association configuration.

reconProgressStateUpdateInterval

integer, optional

Overrides the number of reconciliation operations required before the reconciliation progress state statistics are persisted to the repository. A value of 50 will write statistics to the repository every 50 operations.

Default : 1024, minimum : 1.

reconSourceQueryPageSize

integer

Sets the page size for reconciliation source queries, if paging is enabled.

Default : 10000

reconSourceQueryPaging

boolean, true or false

Specifies whether paging should be used for reconciliation source queries.

Default for non-clustered reconciliation : false
Default for clustered reconciliation : true

reconTargetQueryPageSize

integer

Sets the page size for reconciliation target queries, if paging is enabled.

Default : 10000

reconTargetQueryPaging

boolean, true or false

Specifies whether paging should be used for reconciliation target queries.

Default for non-clustered reconciliation : false
Default for clustered reconciliation : true

result

script object, optional

A script executed after a reconciliation finishes.

The variables available to a result script are as follows:

`source`	Provides statistics about the source phase of the reconciliation
`target`	Provides statistics about the target phase of the reconciliation
`global`	Provides statistics about the entire reconciliation operation
`context`	Information related to the current operation, such as source and target.
`mappingConfig`	A configuration object representing the mapping being processed.
`reconState`	Provides the state of reconciliation operation; such as, success, failure, or active.

runTargetPhase

boolean, true or false

Specifies whether reconciliation operations should run both the source and target phase. To avoid queries on the target resource, set to false.

Default : true

source

string, required

Specifies the path of the source object set. Example: "managed/user".

sourceCondition

script object or queryFilter string, optional

A script or query filter that determines if a source object should be included in the mapping. If no sourceCondition element (or validSource script) is specified, all source objects are included in the mapping.

sourceIdsCaseSensitive

boolean, true or false

Consider case sensitivity when linking source IDs. Only effective if this mapping defines links, ignored if the mapping re-uses another mapping’s links.

Default : true

sourceQueryFullEntry

boolean, true or false, optional

Specifies whether the defined source query returns full object data (true) or IDs only (false).

No default. If not set in the configuration, IDM will attempt to auto-detect the setting, based on the query results.

syncAfter

list of strings, optional

The specified mapping must be synchronized after all mappings in this list.

target

string, required

Specifies the path of the target object set. Example: "system/ldap/account".

targetIdsCaseSensitive

boolean, true or false

Consider case sensitivity when linking target IDs. Only effective if this mapping defines links, ignored if the mapping re-uses another mapping’s links.

Default : true

targetQueryFullEntry

Boolean true or false, optional

Specifies whether the defined target query returns full object data (true) or IDs only (false).

No default. If not set in the configuration, IDM will attempt to auto-detect the setting, based on the query results.

taskThreads

integer, optional

Sets the number of threads dedicated to the same reconciliation run.

Default : 10

triggerSyncProperties

list, optional

A list of JsonPointers to fields in the source object whose changes should trigger a synchronization operation.

validSource

script object, optional

A script that determines if a source object is valid to be mapped. The script yields a boolean value: true indicates the source object is valid; false can be used to defer mapping until some condition is met. In the root scope, the source object is provided in the source property. If the script is not specified, then all source objects are considered valid.

validTarget

script object, optional

A script used during the target phase of reconciliation that determines if a target object is valid to be mapped. The script yields a boolean value: true indicates that the target object is valid; false indicates that the target object should not be included in reconciliation. In the root scope, the target object is provided in the target property. If the script is not specified, then all target objects are considered valid for mapping.

Property objects

A property object specifies how the value of a target property is determined.

{
  "target" : string,
  "source" : string,
  "transform" : script object,
  "condition" : script object,
  "default": value
}

Property object properties

target: string, required

Specifies the path of the property in the target object to map to.
source: string, optional

Specifies the path of the property in the source object to map from. If not specified, then the target property value is derived from the script or default value.
transform: script object, optional

A script to determine the target property value. The root scope contains the value of the source in the source property, if specified. If the source property has a value of "", the entire source object of the mapping is contained in the root scope. The resulting value yielded by the script is stored in the target property.
condition: script object, optional

A script to determine whether the mapping should be executed or not. The condition has an "object" property available in root scope, which (if specified) contains the full source object. For example "source": "(object.email != null)". The script is considered to return a boolean value.
default: any value, optional

Specifies the value to assign to the target property if a non-null value is not established by source or transform. If not specified, the default value is null.

Policy objects

A policy object specifies a link condition and the associated actions to take in response.

{
  "condition"  : optional, script object,
  "situation"  : string,
  "action"     : string or script object
  "postAction" : optional, script object
}

Policy object properties

condition

script object or queryFilter condition, optional

Applies a policy, based on the link type, for example "condition" : "/linkQualifier eq \"user\"".

A queryFilter condition can be expressed in two ways—as a string ("condition" : "/linkQualifier eq \"user\"") or a map, for example:

"condition" : {
    "type" : "queryFilter",
    "filter" : "/linkQualifier eq \"user\""
}

It is generally preferable to express a queryFilter condition as a map.

A condition script has the following variables available in its scope: object and linkQualifier.

situation

string, required

Specifies the situation for which an associated action is to be defined.

action

string or script object, required

Specifies the action to perform. If a script is specified, the script is executed and is expected to yield a string containing the action to perform.

The action script has the following variables available in its scope: source, target, sourceAction, linkQualifier, and recon.

postAction

script object, optional

Specifies the action to perform after the previously specified action has completed.

The postAction script has the following variables available in its scope: source, target, action, sourceAction, linkQualifier, and reconID. sourceAction is true if the action was performed during the source reconciliation phase, and false if the action was performed during the target reconciliation phase. For more information, refer to How Synchronization Situations Are Assessed.

No postAction script is triggered if the action is either IGNORE or ASYNC.

Script Object

Script objects take the following form.

{
  "type"  : "text/javascript",
  "source": string
}

type: string, required

The script type.

IDM supports "text/javascript" and "groovy".
source: string, required

Specifies the source code of the script to be executed.

Links

By default, links are maintained between source and target objects in mappings. This behavior is governed by the enableLinking property in the mapping. If enableLinking is set to false, links are not maintained.

You might want to disable linking in the case of a bulk import or during data migration.

To maintain links between source and target objects in mappings, IDM stores an object set in the repository. The object set identifier follows this scheme:

links/mapping

Here, mapping represents the name of the mapping for which links are managed.

Link entries have the following structure:

{
   "_id":string,
   "_rev":string,
   "linkType":string,
   "firstId":string
   "secondId":string,
}

_id: string

The identifier of the link object.
_rev: string, required

The value of link object’s revision.
linkType: string, required

The type of the link. Usually the name of the mapping which created the link.
firstId: string, required

The identifier of the first of the two linked objects.
secondId: string

The identifier of the second of the two linked objects.

Queries

IDM performs the following queries on a link object set:

Find link(s) for a given firstId object identifier.
```
SELECT * FROM links WHERE linkType = value AND firstId = value
```
Although a single result makes sense, this query is intended to allow multiple results so that this scenario can be handled as an exception.
Select link(s) for a given second object identifier.
```
SELECT * FROM links WHERE linkType = value AND secondId = value
```
Although a single result makes sense, this query is intended to allow multiple results so that this scenario can be handled as an exception.

Reconciliation stages

IDM performs reconciliation on a per-mapping basis. The process of reconciliation for a given mapping includes these stages:

Iterate through all objects for the object set specified as source. For each source object, carry out the following steps.
1. Look for a link to a target object in the link object set, and perform a correlation query (if defined).
2. Determine the link condition, as well as whether a target object can be found.
3. Determine the action to perform based on the policy defined for the condition.
4. Perform the action.
5. Keep track of the target objects for which a condition and action has already been determined.
6. Write the results.
Iterate through all object identifiers for the object set specified as target. For each identifier, carry out the following steps:
1. Find the target in the link object set.
  
  Determine if the target object was handled in the first phase.
2. Determine the action to perform based on the policy defined for the condition.
3. Perform the action.
4. Write the results.
Iterate through all link objects, carrying out the following steps.
1. If the reconId is "my", then skip the object.
  
  If the reconId is not recognized, then the source or the target is missing.
2. Determine the action to perform based on the policy.
3. Perform the action.
4. Store the reconId identifer in the mapping to indicate that it was processed in this run.

To optimize a reconciliation operation, the reconciliation process does not attempt to correlate source objects to target objects if the set of target objects is empty when the correlation is started. For information on changing this default behaviour, refer to Correlate Empty Target Sets.

REST API

External synchronized objects expose an API to request immediate synchronization. This API includes the following requests and responses.

Request

Example:

POST /openidm/system/csvfile/account/jsmith?_action=liveSync HTTP/1.1

Response (success)

Example:

HTTP/1.1 204 No Content
...

Response (synchronization failure)

Example:

HTTP/1.1 409 Conflict
...
[JSON representation of error]

Reconciliation duration metrics

Obtaining the Details of a Reconciliation describes how to obtain the details of a reconciliation run over REST. This section provides more information on the metrics returned when you query the recon endpoint. Reconciliation is processed as a series of distinct tasks. The durationSummary property indicates the period of time spent on each task. You can use this information to address reconciliation performance bottlenecks.

The following sample output shows the kind of information returned for each reconciliation run:

{
    "_id": "3bc72717-a4bb-4871-b936-3a5a560c1a7c-37",
    "duration": 781561,
    "durationSummary": {
        "auditLog": {
            ...
        },
        ...
        "sourceObjectQuery": {
            "count": 100,
            "max": 96,
            "mean": 14,
            "min": 6,
            "stdDev": 16,
            "sum": 1450
        },
        "sourcePagePhase": {
            "count": 1,
            "max": 20944,
            "mean": 20944,
            "min": 20944,
            "stdDev": 0,
            "sum": 20944
        },
        "sourceQuery": {
            "count": 1,
            "max": 120,
            "mean": 120,
            "min": 120,
            "stdDev": 0,
            "sum": 120
        },
        "targetPhase": {
            "count": 1,
            "max": 0,
            "mean": 0,
            "min": 0,
            "stdDev": 0,
            "sum": 0
        },
        "targetQuery": {
            "count": 1,
            "max": 19657,
            "mean": 19657,
            "min": 19657,
            "stdDev": 0,
            "sum": 19657
        }
    },
    ...
}

The specific reconciliation tasks that are run depend on the configuration for that mapping. For example, the sourcePagePhase is run only if paging is enabled. The linkQuery is run only for non-clustered reconciliation operations, because an initial query of all links does not make sense if a single source page query is being run.

Recon tasks

The following list describes all the possible tasks that can be run for a single reconciliation:

sourcePhase

This phase runs only for non-clustered, non-paged reconciliations. The total duration (sum) is the time spent processing all records on the source system.

sourcePagePhase

Queries and processes individual objects in a page, based on their IDs. This phase is run only for clustered reconciliations or for non-clustered reconciliations that have source paging configured. The total duration (sum) is the total time spent processing source pages across all cluster nodes. This processing occurs in parallel across all cluster nodes, so it is normal for the sourcePagePhase duration to exceed the total reconciliation duration.

sourceQuery

Obtains all IDs on the source system, or in a specific source page.

When the sourceQuery returns a null paging cookie, indicating that there are no more source IDs to reconcile, the clustered reconciliation process dispatches a scheduled job named sourcePageCompletionCheck.

This job checks for remaining source page jobs on the scheduler. If there are no remaining source page jobs, the sourcePageCompletionCheck schedules the target phase. If there are still source page jobs to process, the sourcePageCompletionCheck schedules another instance of itself to perform these checks again after a few seconds.

Because the target phase reconciles all IDs that were not reconciled during the source phase, it cannot start until all of the source pages are complete. Final reconciliation statistics cannot be generated and logged until all source page jobs have completed, so the sourcePageCompletionCheck runs even if the target phase is not enabled.

sourceObjectQuery

Queries the individual objects on the source system or page, based on their IDs.

validSourceScript

Processes any scripts that should be run to determine if a source object is valid to be mapped.

linkQuery

Queries any existing links between source and target objects.

This phase includes the following tasks:

sourceLinkQuery: Queries any existing links from source objects to target objects.
targetLinkQuery: Queries any existing links from target objects that were not processed during the sourceLinkQuery phase.
linkQualifiersScript: Runs any link qualifier scripts. For more information, refer to Map a Single Source Object to Multiple Target Objects.
onLinkScript: Processes any scripts that should be run when source and target objects are linked.
onUnlinkScript: Processes any scripts that should be run when source and target objects are unlinked.
deleteLinkObject: Deletes any links that are no longer relevant between source and target objects.

correlationQuery

Processes any configured correlation queries. For more information, refer to Writing Correlation Queries.

correlationScript

Processes any configured correlation scripts. For more information, refer to Writing Correlation Scripts.

onMappingScript

For roles, processes the script that applies the effective assignments as part of the mapping.

activePolicyScript

Sets the action and active policy based on the current situation.

activePolicyPostActionScript

Processes any scripts configured to run after policy validation.

targetPhase

The aggregated result for time spent processing records on the target system.

targetQuery

Queries all IDs on the target system. The list of IDs is restricted to IDs that have not already been linked to a source ID during the source phase. The target query generates a list of orphan IDs that must be reconciled if the target phase is not disabled.

targetObjectQuery

Queries the individual objects on the target system, based on their IDs.

validTargetScript

Processes any scripts that should be run to determine if a target object is valid to be mapped.

onCreateScript

Processes any scripts that should be run when a new target object is created.

updateTargetObject

Updates existing linked target objects, based on the configured situations and actions.

onUpdateScript

Processes any scripts that should be run when a target object is updated.

deleteTargetObject

Deletes any objects on the target resource that must be removed in accordance with the defined synchronization actions.

onDeleteScript

Processes any scripts that should be run when a target object is deleted.

resultScript

Processes the script that is executed when a reconciliation process has finished.

propertyMappingScript

Runs any scripts configured for when source and target properties are mapped.

postMappingScript

Processes any scripts that should be run when synchronization has been performed on the managed/user object.

onReconScript

Processes any scripts that should be run after source and target systems are reconciled.

auditLog

Writes reconciliation results to the audit log.

Metrics Collected

For each phase, the following metrics are collected:

count

The number of objects or records processed during that phase. For the sourcePageQuery phase, the count parameter refers to the page size.

When the count statistic of a particular task refers to the number of records being reconciled, the sum statistic of that task represents the total time across the total number of threads running in all nodes in the cluster. For example:

"updateTargetObject": {
    "count": 1000000,
    "max": 1193,
    "mean": 35,
    "min": 11,
    "stdDev": 0,
    "sum": 35065991
}

max

The maximum time, in milliseconds, spent processing a record during that phase.

mean

The average time, in milliseconds, spent processing a record during that phase.

min

The minimum time, in milliseconds, spent processing a record during that phase.

stdDev

The standard deviation, which measures the variance of the individual values from the mean.

sum

The total amount of time, in milliseconds, spent during that phase.

PingIDM 7.5.0

Synchronization reference

Object-mapping objects

Property objects

Policy objects

Script Object

Links

Queries

Reconciliation stages

REST API

Reconciliation duration metrics

Recon tasks

Metrics Collected