Synchronization reference
The synchronization engine is one of the core PingIDM services. You configure the synchronization service through a mappings
property that specifies mappings between objects that are managed by the synchronization engine.
{
"mappings": [ object-mapping object, ... ]
}
Object-mapping objects
An object-mapping object specifies the configuration for a mapping of source objects to target objects. The name
, source
, and target
properties are mandatory. Other properties are optional or implicit. They have a default value if not set.
{
"correlationQuery" : script object,
"correlationScript" : script object,
"displayName" : string,
"enableLinking" : boolean,
"enableSync" : boolean,
"linkQualifiers" : [ list of strings ] or script object,
"links" : string,
"name" : string,
"onCreate" : script object,
"onDelete" : script object,
"onLink" : script object,
"onMapping" : script object,
"onUnlink" : script object,
"onUpdate" : script object,
"optimizeAssignmentSync" : boolean,
"policies" : [ policy object, ... ],
"postMapping" : script object,
"properties" : [ property object, ... ],
"queuedSync" : { property object },
"reconProgressStateUpdateInterval" : integer,
"reconSourceQueryPageSize" : integer,
"reconSourceQueryPaging" : boolean,
"reconTargetQueryPageSize" : integer,
"reconTargetQueryPaging" : boolean,
"result" : script object,
"runTargetPhase" : boolean,
"source" : string,
"sourceCondition" : script object or queryFilter string,
"sourceIdsCaseSensitive" : boolean,
"sourceQuery" : [ list of properties ],
"sourceQueryFullEntry" : boolean,
"syncAfter" : [ list of strings ],
"target" : string,
"targetIdsCaseSensitive" : boolean,
"targetQuery" : [ list of properties ],
"targetQueryFullEntry" : boolean,
"taskThreads" : integer,
"triggerSyncProperties" : [ list of JSON pointers ],
"validSource" : script object,
"validTarget" : script object
}
Mapping Object Properties
- correlationQuery
-
script object, optional
A script that yields a query object to query the target object set when a source object has no linked target. The syntax for writing the query depends on the target system of the correlation. Learn more in Writing Correlation Queries. The source object is provided in the
source
property in the script scope. - correlationScript
-
script object, optional
A script that goes beyond a
correlationQuery
of a target system. Used when you need another method to determine which records in the target system relate to the given source record. The syntax depends on the target of the correlation. Learn more about defining correlation scripts in Writing Correlation Scripts. - displayName
-
string, optional
The mapping name displayed in the UI.
- enableLinking
-
boolean, true or false
Specifies whether links should be maintained between source and target objects for a mapping.
Default :
true
- enableSync
-
boolean, true or false
Specifies whether automatic synchronization, such as liveSync and implicit synchronization, should be enabled for a specific mapping. Learn more in Disable Automatic Synchronization Operations.
Default :
true
- linkQualifiers
-
list of strings or script object, optional
Enables mapping of a single source object to multiple target objects.
Example:
"linkQualifiers" : [ "employee", "customer" ]
or"linkQualifiers" : { "type" : "text/javascript", "globals" : { }, "source" : "if (returnAll) { ['contractor', 'employee', 'customer', 'manager'] } else { if(object.type === 'employee') { ['employee', 'customer', 'manager'] } else { ['contractor', 'customer'] } }" }
If a script object, the script must return a list of strings.
- links
-
string, optional
Enables reuse of the links created in another mapping. Example:
"systemLdapAccounts_managedUser"
reuses the links created by a previous mapping whosename
is"systemLdapAccounts_managedUser"
. - name
-
string, required
Uniquely names the object mapping. Used in the link object identifier.
- onCreate
-
script object, optional
A script to run when a target object is to be created after property mappings have been applied. In the root scope, the source object is provided in the
source
property, the projected target object in thetarget
property, and the link situation that led to the create operation in thesituation
property. Properties on the target object can be modified by the script. If a property value isn’t set by the script, IDM falls back on the default property mapping configuration. If the script throws an exception, the target object creation is aborted. - onDelete
-
script object, optional
A script to run when a target object is to be deleted, after property mappings have been applied. In the root scope, the source object is provided in the
source
property, the target object in thetarget
property, and the link situation that led to the delete operation in thesituation
property. If the script throws an exception, the target object deletion is aborted. - onLink
-
script object, optional
A script to run when a source object is to be linked to a target object, after property mappings have been applied. In the root scope, the source object is provided in the
source
property and the projected target object in thetarget
property.Although an
onLink
script has access to a copy of the target object, changes made to that copy won’t be saved to the target system automatically. To persist changes made to target objects by anonLink
script, you must explicitly include a call to the action that should be taken on the target object. For example,openidm.create
,openidm.update
oropenidm.delete
) within the script.In the following example, when an LDAP target object is linked, the
"description"
attribute of that object is updated with the value"Active Account"
. A call toopenidm.update
is made within theonLink
script to set the value."onLink" : { "type" : "text/javascript", "source" : "target.description = 'Active Account'; openidm.update('system/ldap/account/' + target._id, null, target);" }
If the script throws an exception, target object linking is aborted.
- onMapping
-
script object, optional
A script that is run as a part of a mapping. For example, a script that applies effective assignments as a part of the mapping. This is run after the mappings have been applied, but before
onUpdate
scripts are run. - onUnlink
-
script object, optional
A script to run when a source and a target object are to be unlinked after property mappings have been applied. In the root scope, the source object is provided in the
source
property and the target object in thetarget
property.Although an
onUnlink
script has access to a copy of the target object, changes made to that copy won’t be saved to the target system automatically. To persist changes made to target objects by anonUnlink
script, you must explicitly include a call to the action that should be taken on the target object. For example,openidm.create
,openidm.update
oropenidm.delete
within the script.In the following example, when an LDAP target object is unlinked, the
description
attribute of that object is updated with the valueInactive Account
. A call toopenidm.update
is made within theonUnlink
script to set the value."onUnlink" : { "type" : "text/javascript", "source" : "target.description = 'Inactive Account'; openidm.update('system/ldap/account/' + target._id, null, target);" }
If the script throws an exception, target object unlinking is aborted.
- onUpdate
-
script object, optional
A script to run when a target object is to be updated after property mappings have been applied. In the root scope, the source object is provided in the
source
property, the projected target object in thetarget
property, and the link situation that led to the update operation in thesituation
property. Any changes that the script makes to the target object will be persisted when the object is finally saved to the target resource. If the script throws an exception, the target object update is aborted. - optimizeAssignmentSync
-
boolean
When set to
true
, a synchronization event is triggered only by modifications to assignments if those modifications are relevant to that particular mapping. When set tofalse
, the default, a synchronization event is triggered for all members of a role or assignment if the relationships, or attributes for an assignment are modified.Don’t include
effectiveAssignments
intriggerSyncProperties
, which negates all performance improvements provided byoptimizeAssignmentSync
. - policies
-
array of policy objects, optional
Specifies a set of link conditions and associated actions to take in response.
- postMapping
-
script object, optional
A script to run when sync has been performed on a managed object. This is run after the source object has been successfully synchronized with the target system.
- properties
-
array of property-mapping objects, optional
Specifies mappings between source object properties and target object properties with optional transformation scripts. Learn more in Property Object Properties.
- queuedSync
-
list of properties, optional
Specifies the queued synchronization configuration.
- reconProgressStateUpdateInterval
-
integer, optional
Overrides the number of reconciliation operations required before the reconciliation progress state statistics are persisted to the repository. A value of
50
will write statistics to the repository for every50
operations.Default :
1024
, minimum :1
- reconSourceQueryPageSize
-
integer
Sets the page size for reconciliation source queries if paging is enabled.
Default for clustered reconciliation :
10000
- reconSourceQueryPaging
-
boolean, true or false
Specifies whether paging should be used for reconciliation source queries.
-
Default for non-clustered reconciliation :
false
-
Default for clustered reconciliation :
true
-
- reconTargetQueryPageSize
-
integer
Sets the page size for reconciliation target queries if paging is enabled.
Default :
10000
- reconTargetQueryPaging
-
boolean, true or false
Specifies whether paging should be used for reconciliation target queries.
-
Default for non-clustered reconciliation :
false
-
Default for clustered reconciliation :
true
-
- result
-
script object, optional
A script run after a reconciliation finishes.
The variables available to a
result
script include:Variable Description source
Provides statistics about the source phase of the reconciliation.
target
Provides statistics about the target phase of the reconciliation.
global
Provides statistics about the entire reconciliation operation.
context
Information related to the current operation, such as source and target.
mappingConfig
A configuration object representing the mapping being processed.
reconState
Provides the state of reconciliation operation, such as success, failure, or active.
- runTargetPhase
-
boolean, true or false
Specifies whether reconciliation operations should run both the source and target phase. To avoid queries on the target resource, set to
false
.Default :
true
- source
-
string, required
Specifies the path of the source objects. Example:
"managed/realm-name_user"
. - sourceCondition
-
script object or
queryFilter
string, optionalA script or
queryFilter
that determines if a source object should be included in the mapping. If nosourceCondition
element orvalidSource
script is specified, all source objects are included in the mapping. - sourceIdsCaseSensitive
-
boolean, true or false
Consider case sensitivity when linking source IDs. Only effective if this mapping defines links and ignored if the mapping reuses links to another mapping.
Default :
true
- sourceQuery
-
list of properties, optional
Specifies the
queryFilter
to be used during reconciliation when querying the source objects. Learn more in Restrict reconciliation by using queries. - sourceQueryFullEntry
-
boolean, true or false, optional
Specifies whether the defined source query returns full object data (
true
) or IDs only (false
).No default. If not set in the configuration, IDM will attempt to auto-detect the setting, based on the query results.
- syncAfter
-
list of strings, optional
The specified mapping must be synchronized after all mappings in this list.
- target
-
string, required
Specifies the path of the target objects. Example:
"system/ldap/account"
. - targetIdsCaseSensitive
-
boolean, true or false
Consider case sensitivity when linking target IDs. Only effective if this mapping defines links and ignored if the mapping reuses links.
Default :
true
- targetQuery
-
list of properties, optional
Specifies the
queryFilter
to be used during reconciliation when querying the target objects. Learn more in Restrict reconciliation by using queries. - targetQueryFullEntry
-
Boolean
true
orfalse
, optionalSpecifies whether the defined target query returns full object data (
true
) or IDs only (false
).No default. If not set in the configuration, IDM will attempt to auto-detect the setting based on the query results.
- taskThreads
-
integer, optional
Sets the number of threads dedicated to the same reconciliation run.
Default :
10
- triggerSyncProperties
-
list, optional
A list of
JsonPointers
to fields in the source object whose changes should trigger a synchronization operation. - validSource
-
script object, optional
A script that determines if a source object is valid to be mapped. The script yields a boolean value:
true
indicates the source object is valid andfalse
can be used to defer mapping until some condition is met. In the root scope, the source object is provided in thesource
property. If the script isn’t specified, then all source objects are considered valid. - validTarget
-
script object, optional
A script used during the target phase of reconciliation that determines if a target object is valid to be mapped. The script yields a boolean value:
true
indicates that the target object is valid andfalse
indicates that the target object shouldn’t be included in reconciliation. In the root scope, the target object is provided in thetarget
property. If the script isn’t specified, then all target objects are considered valid for mapping.
Property Objects
A property object specifies how the value of a target property is determined.
{
"target" : string,
"source" : string,
"transform" : script object,
"condition" : script object,
"default": value
}
- target
-
string, required
Specifies the path of the property in the target object to map to.
- source
-
string, optional
Specifies the path of the property in the source object to map from. If not specified, then the target property value is derived from the script or default value.
- transform
-
script object, optional
A script to determine the target property value. The root scope contains the value of the source in the
source
property, if specified. If thesource
property has a value of""
, the entire source object of the mapping is contained in the root scope. The resulting value yielded by the script is stored in the target property. - condition
-
script object, optional
A script to determine whether the mapping should be executed or not. The condition has an
"object"
property available in root scope, which (if specified) contains the full source object. For example"source": "(object.email != null)"
. The script is considered to return a boolean value. - default
-
any value, optional
Specifies the value to assign to the target property if a non-null value isn’t established by
source
ortransform
. If not specified, the default value isnull
.
Policy Objects
A policy object specifies a link condition and the associated actions to take in response.
{
"condition" : optional, script object,
"situation" : string,
"action" : string or script object
"postAction" : optional, script object
}
- condition
-
script object or queryFilter condition, optional
Applies a policy, based on the link type, for example
"condition" : "/linkQualifier eq \"user\""
.A queryFilter condition can be expressed in two ways—as a string (
"condition" : "/linkQualifier eq \"user\""
) or a map, for example:"condition" : { "type" : "queryFilter", "filter" : "/linkQualifier eq \"user\"" }
It is generally preferable to express a queryFilter condition as a map.
A
condition
script has the following variables available in its scope:object
andlinkQualifier
. - situation
-
string, required
Specifies the situation for which an associated action is to be defined.
- action
-
string or script object, required
Specifies the action to perform. If a script is specified, the script is executed and is expected to yield a string containing the action to perform.
The
action
script has the following variables available in its scope:source
,target
,sourceAction
,linkQualifier
, andrecon
. - postAction
-
script object, optional
Specifies the action to perform after the previously specified action has completed.
The
postAction
script has the following variables available in its scope:source
,target
,action
,sourceAction
,linkQualifier
, andreconID
.sourceAction
istrue
if the action was performed during the source reconciliation phase, andfalse
if the action was performed during the target reconciliation phase. For more information, refer to How Synchronization Situations Are Assessed.No postAction
script is triggered if theaction
is eitherIGNORE
orASYNC
.
Links
By default, links are maintained between source and target objects in mappings. This behavior is governed by the enableLinking
property in the mapping. If enableLinking
is set to false
, links are not maintained.
You may want to disable linking in the case of a bulk import.
To maintain links between source and target objects in mappings, IDM stores an object set in the repository. The object set identifier follows this scheme:
links/mapping
Here, mapping represents the name of the mapping for which links are managed.
Link entries have the following structure:
{
"_id":string,
"_rev":string,
"linkType":string,
"firstId":string
"secondId":string,
}
- _id
-
string
The identifier of the link object.
- _rev
-
string, required
The value of link object’s revision.
- linkType
-
string, required
The type of the link. Usually the name of the mapping which created the link.
- firstId
-
string, required
The identifier of the first of the two linked objects.
- secondId
-
string
The identifier of the second of the two linked objects.
Queries
IDM performs the following queries on a link object set:
-
Find link(s) for a given firstId object identifier.
SELECT * FROM links WHERE linkType = value AND firstId = value
Although a single result makes sense, this query is intended to allow multiple results so that this scenario can be handled as an exception.
-
Select link(s) for a given second object identifier.
SELECT * FROM links WHERE linkType = value AND secondId = value
Reconciliation Stages
IDM performs reconciliation on a per-mapping basis. The process of reconciliation for a given mapping includes these stages:
-
Iterate through all objects for the object set specified as
source
. For each source object, carry out the following steps.-
Look for a link to a target object in the link object set, and perform a correlation query (if defined).
-
Determine the link condition, as well as whether a target object can be found.
-
Determine the action to perform based on the policy defined for the condition.
-
Perform the action.
-
Keep track of the target objects for which a condition and action has already been determined.
-
Write the results.
-
-
Iterate through all object identifiers for the object set specified as
target
. For each identifier, carry out the following steps:-
Find the target in the link object set.
Determine if the target object was handled in the first phase.
-
Determine the action to perform based on the policy defined for the condition.
-
Perform the action.
-
Write the results.
-
-
Iterate through all link objects, carrying out the following steps.
-
If the
reconId
is"my"
, then skip the object.If the
reconId
is not recognized, then the source or the target is missing. -
Determine the action to perform based on the policy.
-
Perform the action.
-
Store the
reconId
identifer in the mapping to indicate that it was processed in this run.
-
To optimize a reconciliation operation, the reconciliation process does not attempt to correlate source objects to target objects if the set of target objects is empty when the correlation is started. For information on changing this default behaviour, refer to Correlate Empty Target Sets. |
REST API
External synchronized objects expose an API to request immediate synchronization. This API includes the following requests and responses.
- Request
-
Example:
POST /openidm/system/csvfile/account/jsmith?_action=liveSync HTTP/1.1
- Response (success)
-
Example:
HTTP/1.1 204 No Content ...
- Response (synchronization failure)
-
Example:
HTTP/1.1 409 Conflict ... [JSON representation of error]
Reconciliation Duration Metrics
Obtaining the Details of a Reconciliation describes how to obtain the details of a reconciliation run over REST. This section provides more information on the metrics returned when you query the recon
endpoint. Reconciliation is processed as a series of distinct tasks. The durationSummary
property indicates the period of time spent on each task. You can use this information to address reconciliation performance bottlenecks.
The following sample output shows the kind of information returned for each reconciliation run:
{
"_id": "3bc72717-a4bb-4871-b936-3a5a560c1a7c-37",
"duration": 781561,
"durationSummary": {
"auditLog": {
...
},
...
"sourceObjectQuery": {
"count": 100,
"max": 96,
"mean": 14,
"min": 6,
"stdDev": 16,
"sum": 1450
},
"sourcePagePhase": {
"count": 1,
"max": 20944,
"mean": 20944,
"min": 20944,
"stdDev": 0,
"sum": 20944
},
"sourceQuery": {
"count": 1,
"max": 120,
"mean": 120,
"min": 120,
"stdDev": 0,
"sum": 120
},
"targetPhase": {
"count": 1,
"max": 0,
"mean": 0,
"min": 0,
"stdDev": 0,
"sum": 0
},
"targetQuery": {
"count": 1,
"max": 19657,
"mean": 19657,
"min": 19657,
"stdDev": 0,
"sum": 19657
}
},
...
}
The specific reconciliation tasks that are run depend on the configuration for that mapping. For example, the sourcePagePhase
is run only if paging is enabled. The linkQuery
is run only for non-clustered reconciliation operations, because an initial query of all links does not make sense if a single source page query is being run.
Recon Tasks
The following list describes all the possible tasks that can be run for a single reconciliation:
sourcePhase
-
This phase runs only for non-clustered, non-paged reconciliations. The total duration (
sum
) is the time spent processing all records on the source system. sourcePagePhase
-
Queries and processes individual objects in a page, based on their IDs. This phase is run only for clustered reconciliations or for non-clustered reconciliations that have source paging configured. The total duration (
sum
) is the total time spent processing source pages across all cluster nodes. This processing occurs in parallel across all cluster nodes, so it is normal for thesourcePagePhase
duration to exceed the total reconciliation duration. sourceQuery
-
Obtains all IDs on the source system, or in a specific source page.
When the
sourceQuery
returns a null paging cookie, indicating that there are no more source IDs to reconcile, the clustered reconciliation process dispatches a scheduled job namedsourcePageCompletionCheck
.This job checks for remaining source page jobs on the scheduler. If there are no remaining source page jobs, the
sourcePageCompletionCheck
schedules the target phase. If there are still source page jobs to process, thesourcePageCompletionCheck
schedules another instance of itself to perform these checks again after a few seconds.Because the target phase reconciles all IDs that were not reconciled during the source phase, it cannot start until all of the source pages are complete. Final reconciliation statistics cannot be generated and logged until all source page jobs have completed, so the
sourcePageCompletionCheck
runs even if the target phase is not enabled. sourceObjectQuery
-
Queries the individual objects on the source system or page, based on their IDs.
validSourceScript
-
Processes any scripts that should be run to determine if a source object is valid to be mapped.
linkQuery
-
Queries any existing links between source and target objects.
This phase includes the following tasks:
sourceLinkQuery
-
Queries any existing links from source objects to target objects.
targetLinkQuery
-
Queries any existing links from target objects that were not processed during the
sourceLinkQuery
phase. linkQualifiersScript
-
Runs any link qualifier scripts. For more information, refer to Map a Single Source Object to Multiple Target Objects.
onLinkScript
-
Processes any scripts that should be run when source and target objects are linked.
onUnlinkScript
-
Processes any scripts that should be run when source and target objects are unlinked.
deleteLinkObject
-
Deletes any links that are no longer relevant between source and target objects.
correlationQuery
-
Processes any configured correlation queries. For more information, refer to Writing Correlation Queries.
correlationScript
-
Processes any configured correlation scripts. For more information, refer to Writing Correlation Scripts.
onMappingScript
-
For roles, processes the script that applies the effective assignments as part of the mapping.
activePolicyScript
-
Sets the action and active policy based on the current situation.
activePolicyPostActionScript
-
Processes any scripts configured to run after policy validation.
targetPhase
-
The aggregated result for time spent processing records on the target system.
targetQuery
-
Queries all IDs on the target system. The list of IDs is restricted to IDs that have not already been linked to a source ID during the source phase. The target query generates a list of orphan IDs that must be reconciled if the target phase is not disabled.
targetObjectQuery
-
Queries the individual objects on the target system, based on their IDs.
validTargetScript
-
Processes any scripts that should be run to determine if a target object is valid to be mapped.
onCreateScript
-
Processes any scripts that should be run when a new target object is created.
updateTargetObject
-
Updates existing linked target objects, based on the configured situations and actions.
onUpdateScript
-
Processes any scripts that should be run when a target object is updated.
deleteTargetObject
-
Deletes any objects on the target resource that must be removed in accordance with the defined synchronization actions.
onDeleteScript
-
Processes any scripts that should be run when a target object is deleted.
resultScript
-
Processes the script that is executed when a reconciliation process has finished.
propertyMappingScript
-
Runs any scripts configured for when source and target properties are mapped.
postMappingScript
-
Processes any scripts that should be run when synchronization has been performed on the
managed/realm-name_user
object. onReconScript
-
Processes any scripts that should be run after source and target systems are reconciled.
auditLog
-
Writes reconciliation results to the audit log.
Metrics Collected
For each phase, the following metrics are collected:
count
-
The number of objects or records processed during that phase. For the
sourcePageQuery
phase, thecount
parameter refers to the page size.When the
count
statistic of a particular task refers to the number of records being reconciled, thesum
statistic of that task represents the total time across the total number of threads running in all nodes in the cluster. For example:"updateTargetObject": { "count": 1000000, "max": 1193, "mean": 35, "min": 11, "stdDev": 0, "sum": 35065991 }
max
-
The maximum time, in milliseconds, spent processing a record during that phase.
mean
-
The average time, in milliseconds, spent processing a record during that phase.
min
-
The minimum time, in milliseconds, spent processing a record during that phase.
stdDev
-
The standard deviation, which measures the variance of the individual values from the mean.
sum
-
The total amount of time, in milliseconds, spent during that phase.