--- title: Filter synchronization data description: Filter and restrict sync data in PingIDM using scripts, queries, specific IDs, and property-change triggers to optimize reconciliation performance component: pingidm version: 8 page_id: pingidm:synchronization-guide:chap-restricting-sync canonical_url: https://docs.pingidentity.com/pingidm/8.1/synchronization-guide/chap-restricting-sync.html llms_txt: https://docs.pingidentity.com/pingidm/llms.txt docs_for_agents: https://developer.pingidentity.com/build-with-ai/docs-for-agents.md keywords: ["Synchronization"] superseded_by: https://docs.pingidentity.com/pingidm/8.1/synchronization-guide/chap-restricting-sync.html section_ids: filtering-source-and-target: Filter source and target objects with scripts recon-by-query: Restrict reconciliation by using queries restricting-recons-ui: Restrict reconciliation queries using the admin UI recon-by-id: Restrict reconciliation to a specific ID recon-by-id-override-params: Override mapping configuration for reconById restricting-implicit-sync: Restrict implicit synchronization to specific property changes --- # Filter synchronization data By default, IDM synchronizes all objects that match those defined in the connector configuration for the resource. Many connectors let you limit the scope of objects that the connector accesses. For example, the LDAP connector lets you specify base DNs and LDAP filters so that you don't need to access every entry in the directory. The following sections describe other ways to filter objects or attributes to restrict the synchronization load. ## Filter source and target objects with scripts Synchronization operations can filter source or target objects using the `validSource`, `validTarget`, or `sourceCondition` properties within a mapping configuration: * `validSource` A script that determines if a source object is valid to be mapped. The script yields a Boolean value: `true` indicates that the source object is valid. A `false` value defers a mapping until some condition is met. In the root scope, the source object is provided in the `source` property. If the script isn't specified, then all source objects are considered valid: ```json { "validSource": { "type": "text/javascript", "source": "source.ldapPassword != null" } } ``` * `validTarget` A script used during the second phase or target phase of reconciliation that determines if a discovered target object is valid for mapping or linking. The script yields a Boolean value: * `true`: The target object is valid for processing in the current context. * `false`: The target object is invalid and should be excluded from processing in the current context. In the root scope, the target object is provided in the `target` property. If a `validTarget` script isn't specified, then all target objects are considered valid by default. **Important considerations:** * **Target phase (primary use):** This script executes against target objects found on the target system that were **not** processed during the source phase, such as target-only records. The possible situations for records are: * `TARGET_IGNORED`: The `validTarget` script determines the target isn't valid. * `UNASSIGNED`: The target is valid and isn't linked to a source object. * `SOURCE_MISSING`: The target is valid and is linked to a source that no longer exists. If your script needs to take an action other than logging, you can use `UNLINK` or `DELETE`. * **Source phase (limited use):** During the source phase, this script is **not executed** for target objects already linked or correlated to a source object. This behavior ensures that source-driven updates proceed and existing links are maintained, even if the target object's current state might return `false` from this script's logic. A `validTarget` doesn't change the situation assessment during the source phase. * **Source phase exception:** The script **is** executed during the source phase specifically when evaluating a target object whose linked source record is missing. This scenario occurs from source record deletions managed through implicit synchronization or liveSync. If logic is needed to conditionally handle updates to **already linked** targets based on the target's state during the source phase, you should implement it within scripted action handlers for relevant situations. Example: ```json { "validTarget": { "type": "text/javascript", "source": "target.employeeType == 'internal'" } } ``` * `sourceCondition` An additional filter that must be met for a source object to be included in a mapping. This condition works like a `validSource` script. Its value can be either a `queryFilter` string or a script configuration. A `sourceCondition` is used to specify that a mapping applies to only a particular role or entitlement. The following `sourceCondition` restricts synchronization to those user objects whose account status is `active`: ```json { "mappings": [ { "name": "managedUser_systemLdapAccounts", "source": "managed/user", "sourceCondition": "/source/accountStatus eq \"active\"", ... } ] } ``` During synchronization, scripts and filters have access to `source` and `target` objects. Examples already shown in this section use `source.attributeName` to retrieve attributes from the source objects. Scripts can also write to target attributes using `target.attributeName` syntax, for example: ```json { "onUpdate": { "type": "text/javascript", "source": "if (source.email != null) {target.mail = source.email;}" } } ``` The `sourceCondition` filter also has the `linkQualifier` variable in its scope. Learn more about scripting in the [Scripting function reference](../scripting-guide/scripting-func-ref.html). ## Restrict reconciliation by using queries A reconciliation operation performs a query on the source and on the target resource to determine which records should be reconciled. The default source and target queries are `_queryFilter=true&_fields=_id`, which means that all records in both the source and the target are considered candidates for that reconciliation operation. You can restrict reconciliation to specific entries by defining an explicit `sourceQuery` or `targetQuery` in the mapping configuration. | | | | - | ----------------------------------------------------------------------------------------------------------------------------- | | | The `sourceQuery` filter is ignored during the target phase, and the `targetQuery` filter is ignored during the source phase. | For example, to restrict reconciliation to those records whose `employeeType` on the source resource is `Permanent`, you might specify a source query as follows: ```json "mappings" : [ { "name" : "managedUser_systemLdapAccounts", "source" : "managed/user", "target" : "system/ldap/account", "sourceQuery" : { "_queryFilter" : "employeeType eq \"Permanent\"" }, ... ``` The format of the query can be any query type that's supported by the resource and can include additional parameters, if applicable. Use the `_queryFilter` parameter in common filter notation. By default, the source and target queries send the query to the resource that's defined for that source or target. You can override the resource the query is sent to by specifying a `resourceName` in the query. For example, to query a specific endpoint instead of the source resource, you could modify the preceding source query as follows: ```json { "mappings" : [ { "name" : "managedUser_systemLdapAccounts", "source" : "managed/user", "target" : "system/ldap/account", "sourceQuery" : { "resourceName" : "endpoint/scriptedQuery" "_queryFilter" : "employeeType eq \"Permanent\"" }, ... } ``` To override a source or target query defined in the mapping, you can specify the query when you call the reconciliation operation. For example, to reconcile all employee entries, and not just the permanent employees, you can run the reconciliation operation as follows: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --header "Content-Type: application/json" \ --request POST \ --data '{"sourceQuery": {"_queryFilter" : "true"}}' \ "http://localhost:8080/openidm/recon?_action=recon&mapping=managedUser_systemLdapAccounts" ``` By default, a reconciliation operation runs both the source and target phase. To avoid queries on the target resource, set `runTargetPhase` to `false` in the mapping configuration. To prevent the target resource from being queried during the reconciliation operation configured in the previous example, amend the mapping configuration as follows: ```json { "mappings" : [ { "name" : "systemLdapAccounts_managedUser", "source" : "system/ldap/account", "target" : "managed/user", "sourceQuery" : { "_queryFilter" : "employeeType eq \"Permanent\"" }, "runTargetPhase" : false, ... ``` ### Restrict reconciliation queries using the admin UI 1. From the navigation bar, click Configure > Mappings. 2. On the Mappings page, select the mapping to restrict. 3. Click the Association tab, and expand the Reconciliation Query Filters node. 4. Create a source or target query and click Save. ## Restrict reconciliation to a specific ID You can restrict reconciliation to a specific record in the same way as you restrict reconciliation by using queries. To restrict reconciliation to a specific ID, use the `reconById` action, instead of the `recon` action. Specify the ID with the `id` parameter. The `reconById` action can reconcile only a single ID at a time. The following command reconciles only user ID `b3c2f414-e7b3-46aa-8ce6-f4ab1e89288c` for the mapping `managedUser_systemLdapAccounts`. The command synchronizes this particular user account in LDAP with the data from the managed user repository. The example assumes implicit synchronization is disabled and that a reconciliation operation is required to copy changes made in the repository to the LDAP system: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request POST \ "http://localhost:8080/openidm/recon?_action=reconById&mapping=managedUser_systemLdapAccounts&id=b3c2f414-e7b3-46aa-8ce6-f4ab1e89288c" ``` Reconciliation by ID takes the default reconciliation options specified in the mapping. As a result, the source and target queries, and source and target phases apply equally to reconciliation by ID. ### Override mapping configuration for `reconById` You can override specific mapping configuration properties for a `reconById` run by including a JSON body in the POST request. The following optional properties are supported: | Property | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sourceQuery` | Overrides the source query defined in the mapping. For `reconById`, IDM always constrains the source query filter to the specified `id`, but other source query properties (such as `resourceName` or `_fields`) can be overridden. | | `targetQuery` | Overrides the target query defined in the mapping. Only used if `runTargetPhase` is `true`. | | `runTargetPhase` | Whether to run the target phase. For `reconById`, this defaults to `false`, unlike a full reconciliation where the default is `true`. | | `allowEmptySourceSet` | Whether to allow reconciliation to proceed when the source query returns no results. Learn more in [Prevent the accidental deletion of a target system](prevent-accidental-deletion.html). | | `sourceQueryFullEntry` | Whether the source query returns full object data (`true`) or only IDs (`false`). If set to `false`, IDM performs an individual read for each source object. If not set, IDM attempts to auto-detect based on query results. | | `targetQueryFullEntry` | Whether the target query returns full object data (`true`) or only IDs (`false`). If set to `false`, IDM performs an individual read for each target object. If not set, IDM attempts to auto-detect based on query results. | For example, to reconcile a specific source ID and skip the target phase: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --header "Content-Type: application/json" \ --request POST \ --data '{ "runTargetPhase": false }' \ "http://localhost:8080/openidm/recon?_action=reconById&mapping=managedUser_systemLdapAccounts&id=b3c2f414-e7b3-46aa-8ce6-f4ab1e89288c" ``` To override the target query for a `reconById` run: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --header "Content-Type: application/json" \ --request POST \ --data '{ "runTargetPhase": true, "targetQuery": { "_queryFilter": "true", "resourceName": "system/ldap/account" } }' \ "http://localhost:8080/openidm/recon?_action=reconById&mapping=managedUser_systemLdapAccounts&id=b3c2f414-e7b3-46aa-8ce6-f4ab1e89288c" ``` ## Restrict implicit synchronization to specific property changes For a mapping that has managed objects as the source, an implicit synchronization is triggered if *any* source property changes, regardless of whether the modified property is explicitly defined as a `source` property in the mapping. This default behavior is helpful in situations where no source properties are explicitly defined. Any property within the object is included as a part of the mapping. However, this behavior adds a processing overhead, because every mapping from the managed object is invoked when *any* managed object property changes. If several mappings are configured from the managed object, this default behavior can cause performance issues. In these situations, you can restrict the properties that should trigger an implicit synchronization per mapping, using the `triggerSyncProperties` attribute. This attribute contains an array of JSON pointers to the properties that must change before an implicit synchronization to the target is triggered. If none of these properties changes, no synchronization is triggered, even if other properties in the object change. In the following example, implicit synchronization is triggered only if the `mail`, `telephoneNumber`, or `userName` of an object changes: ```json { "mappings" : [ { "name" : "managedUser_systemLdapAccounts", "source" : "managed/user", "target" : "system/ldap/account", "enableLinking" : false, "triggerSyncProperties" : [ "/mail", "/telephoneNumber", "/userName" ], "properties" : [], "policies" : [] } ] } ``` If any other property changes on the managed object, no implicit synchronization is triggered.