--- title: Synchronize data from a CSV file to IDM description: This sample demonstrates one-way synchronization from an external resource to an IDM repository. component: pingidm version: 8.1 page_id: pingidm:samples-guide:sync-with-csv canonical_url: https://docs.pingidentity.com/pingidm/8.1/samples-guide/sync-with-csv.html keywords: ["Samples", "Configuration", "Synchronization", "CSV"] section_ids: about-sync-with-csv: Sample overview sync-with-csv-configuration: Sample configuration files prepare-sync-with-csv: Prepare the sample run-sync-with-csv: Run the sample sync-with-csv-cli: Run the sample using the command line sync-with-csv-adminui: Run the sample using the admin UI --- # Synchronize data from a CSV file to IDM This sample demonstrates one-way synchronization from an external resource to an IDM repository. The external resource in this case is a simple CSV file. User objects in that file are synchronized with the managed users in the IDM repository. ## Sample overview IDM connects data objects held in separate resources by mapping one object to another. To connect to external resources, IDM uses *connectors*, that are configured for each external resource. When objects in one external resource change, IDM determines how the changes affect the objects in the connected resource, and can make the changes in that resource as necessary. This sample demonstrates how IDM does this by using *reconciliation*. Reconciliation compares the objects in one resource to the mapped objects in another resource. For a complete explanation of reconciliation and synchronization, refer to [Synchronization types](../synchronization-guide/sync-types.html). In this sample, IDM connects to a CSV file that holds sample user data. The CSV file is configured as the authoritative source. A *mapping* is configured between objects in the CSV file and managed user objects in the IDM repository. Note that you can use IDM to synchronized objects between two external resources without going through the IDM repository. In such a case, objects are synchronized directly through connectors to the external resources. This sample involves only one external resource. In practice, you can connect as many resources as needed for your deployment. ## Sample configuration files The configuration files for this sample are located in the `/path/to/openidm/samples/sync-with-csv/conf` directory. When you start IDM with the `-p` project variable (`./startup.sh -p samples/sync-with-csv`), the *project location* (`&{idm.instance.dir}`) is set to a value of `samples/sync-with-csv`. All subsequent paths use this project location as a base. Throughout this documentation, you will see things like "...in your project's `conf/` directory...". The "project" refers to the value of the `&{idm.instance.dir}` variable. The following configuration files play important roles in this sample: * `samples/sync-with-csv/conf/provisioner.openicf-csvfile.json` This file provides the configuration for this instance of the CSV connector. It describes, among other things, the connector version, the location of the CSV file resource, and the object types that are supported for this connection. For a complete understanding of connector configuration files, refer to [Configure connectors](https://docs.pingidentity.com/openicf/connector-reference/configure-connector.html). * `samples/sync-with-csv/conf/sync.json` This file, also called a *mapping file*, defines the configuration for reconciliation and synchronization. This sample file includes only one mapping - `systemCsvfileAccounts_managedUser`. The mapping specifies the synchronization configuration between the CSV file (source) and the IDM repository (target). Examine the file to see how objects are mapped between the two resources, and the actions that IDM should take when it finds objects in specific situations: ```json { "mappings": [ { "name" : "systemCsvfileAccounts_managedUser", "source" : "system/csvfile/account", "target": "managed/user", "correlationQuery": { "type": "text/javascript", "source": "var query = {'_queryId' : 'for-userName', 'uid' : source.name};query;" }, "properties": [ { "source": "email", "target": "mail" }, { "source": "firstname", "target": "givenName" }, { "source": "lastname", "target": "sn" }, { "source": "description", "target": "description" }, { "source": "_id", "target": "_id" }, { "source": "name", "target": "userName" }, { "source": "password", "target": "password" }, { "source" : "mobileTelephoneNumber", "target" : "telephoneNumber" }, { "source" : "roles", "transform" : { "type" : "text/javascript", "source" : "var _ = require('lib/lodash'); _.map(source.split(','), function(role) { return {'_ref': 'internal/role/' + role} });" }, "target" : "authzRoles" } ], "policies": [ { "situation": "CONFIRMED", "action": "UPDATE" }, { "situation": "FOUND", "action": "IGNORE" }, { "situation": "ABSENT", "action": "CREATE" }, { "situation": "AMBIGUOUS", "action": "IGNORE" }, { "situation": "MISSING", "action": "IGNORE" }, { "situation": "SOURCE_MISSING", "action": "IGNORE" }, { "situation": "UNQUALIFIED", "action": "IGNORE" }, { "situation": "UNASSIGNED", "action": "IGNORE" } ] } ] } ``` Source and target paths that start with `managed`, such as `managed/user`, always refer to objects in the IDM repository. Paths that start with `system`, such as `system/csvfile/account`, refer to external objects, in this case, objects in the CSV file. When you start a reconciliation, IDM queries all users in the source, and then creates, deletes, or modifies users in the IDM repository, as mapped in `conf/sync.json`. For more information about synchronization, reconciliation, and mappings, refer to [Synchronization](../synchronization-guide/preface.html). * `samples/sync-with-csv/conf/schedule-reconcile_systemCsvAccounts_managedUser.json` The sample schedule configuration file defines a task that launches a reconciliation every minute for the mapping named `systemCsvfileAccounts_managedUser`. The schedule is disabled by default: ```json { "enabled" : false, "type": "simple", "repeatInterval": 3600000, "persisted" : true, "concurrentExecution" : false, "misfirePolicy" : "fireAndProceed", "invokeService" : "sync", "invokeContext" : { "action" : "reconcile", "mapping" : "systemCsvfileAccounts_managedUser" } } ``` IDM regularly scans the `conf/` directory for any [schedule configuration](../schedules-guide/schedules.html) files. Apart from the scheduled reconciliation run, you can also start reconciliation run through the REST interface. The call to the REST interface is an HTTP POST such as the following: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request POST \ "http://localhost:8080/openidm/recon?_action=recon&mapping=systemCsvfileAccounts_managedUser&waitForCompletion=true" ``` The `waitForCompletion=true` parameter specifies that the operation should return only when it has completed. * `samples/sync-with-csv/data/csvConnectorData.csv` This CSV file is the external resource or data store in this sample. The file contains two users, bjensen and scarter. During the sample, you will reconcile those users *from* the CSV file *to* the managed user repository. ## Prepare the sample 1. [Set up DS](start-here.html#ldap-server-config) without importing any LDIF file. 2. [Prepare IDM](start-here.html#preparing-openidm), and start the server using the sample configuration: ``` cd /path/to/openidm/ ./startup.sh -p samples/sync-with-csv ``` ## Run the sample You can work through the sample using the command line, or using the admin UI: ### Run the sample using the command line 1. After starting IDM, reconcile the objects in both resources. You can trigger the reconciliation either by setting `"enabled" : true` in the schedule configuration file (`conf/schedule-reconcile_systemCsvAccounts_managedUser.json`) and then waiting until the scheduled reconciliation happens, or by running the following `curl` command: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request POST \ "http://localhost:8080/openidm/recon?_action=recon&mapping=systemCsvfileAccounts_managedUser&waitForCompletion=true" ``` Successful reconciliation returns a reconciliation run ID, and the status of the reconciliation operation, as follows: ``` { "_id":"2d87c817-3d00-4776-a705-7de2c65937d8", "state":"SUCCESS" } ``` 2. Display the managed user records that were created by the reconciliation operation. You can use any REST client to query the repository. Perform an HTTP GET on the URL `"http://localhost:8080/openidm/managed/user?_queryFilter=true"` with the headers `"X-OpenIDM-Username: openidm-admin"` and `"X-OpenIDM-Password: openidm-admin"`. The following example uses the [curl](https://curl.se) command to get all managed user records, in JSON format: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ "http://localhost:8080/openidm/managed/user?_queryFilter=true" { "result": [ { "_id": "bjensen", "_rev": "00000000e17186b6", "mail": "bjensen@example.com", "givenName": "Barbara", "sn": "Jensen", "description": "Created By CSV", "userName": "bjensen", "telephoneNumber": "1234567", "accountStatus": "active", "effectiveAssignments": [], "effectiveRoles": [] }, { "_id": "scarter", "_rev": "00000000970685c3", "mail": "scarter@example.com", "givenName": "Steven", "sn": "Carter", "description": "Created By CSV", "userName": "scarter", "telephoneNumber": "1234567", "accountStatus": "active", "effectiveAssignments": [], "effectiveRoles": [] } ], ... } ``` You can use any query filter to return the information you need. Learn more in [Define and call data queries](../objects-guide/queries.html). 3. Now display the user record for `bjensen` by appending the user ID to the request URL: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ "http://localhost:8080/openidm/managed/user/bjensen" { "_id": "bjensen", "_rev": "00000000e17186b6", "mail": "bjensen@example.com", "givenName": "Barbara", "sn": "Jensen", "description": "Created By CSV", "userName": "bjensen", "telephoneNumber": "1234567", "accountStatus": "active", "effectiveAssignments": [], "effectiveRoles": [] } ``` The request returns the complete user record for `bjensen`. 4. Restrict the query output with the `fields` parameter, as follows: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ "http://localhost:8080/openidm/managed/user?_queryFilter=true&_fields=userName,mail" { "result": [ { "_id": "bjensen", "_rev": "00000000e17186b6", "mail": "bjensen@example.com", "userName": "bjensen" }, { "_id": "scarter", "_rev": "00000000970685c3", "mail": "scarter@example.com", "userName": "scarter" } ], ... } ``` 5. To test the scheduled reconciliation, add a user to the CSV data file, `samples/sync-with-csv/data/csvConnectorData.csv`. For example, add user `jberg` as follows: ```csv "description", "uid", "username", "firstname", "lastname", "email", "mobile... "Created ...", "bjensen", "bjensen@example.com", "Barbara", "Jensen", "bjensen@example.com", "123456... "Created ...", "scarter", "scarter@example.com", "Steven", "Carter", "scarter@example.com", "123456... "Created ...", "jberg", "jberg@example.com", "James", "Berg", "jberg@example.com", "123456... ``` 6. If you enabled the scheduled reconciliation in Step 1, you can simply wait for the reconciliation operation to run. Otherwise, run the reconciliation manually with the same command you used in that step. 7. After the reconciliation has run, query the managed user repository to view the new user in the list of managed users: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ "http://localhost:8080/openidm/managed/user?_queryFilter=true&_fields=_id" { "result": [ { "_id": "bjensen", "_rev": "00000000e17186b6" }, { "_id": "scarter", "_rev": "00000000970685c3" }, { "_id": "jberg", "_rev": "00000000ea628233" } ], ... } ``` 8. To view the reconciliation details, query the reconciliation using its `id`: ``` curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ "http://localhost:8080/openidm/recon/assoc/2d87c817-3d00-4776-a705-7de2c65937d8" { "_id": "2d87c817-3d00-4776-a705-7de2c65937d8", "_rev": "1", "mapping": "systemCsvfileAccounts_managedUser", "sourceResourceCollection": "managed/user", "targetResourceCollection": "system/csv/account", "isAnalysis": "false", "finishTime": "2022-05-01T23:36:24.434153Z" } ``` For more information on reconciliation operations via REST, refer to [Manage reconciliation](../synchronization-guide/manage-recon.html). You configure the action that IDM takes for each situation in the mapping file, `conf/sync.json`. For the list of all possible situations and actions, refer to [Synchronization situations and actions](../synchronization-guide/chap-situations-actions.html). | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------- | | | If you've [enabled audit logging](../audit-guide/audit.html), you can view the reconciliation details in the `openidm/audit/recon.audit.json` file. | ### Run the sample using the admin UI | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | Starting with IDM 8.1, the [legacy admin UI is deprecated](../release-notes/deprecated-functionality.html#legacy-admin-ui-deprecated) and is no longer bundled with IDM. New deployments should use the [Platform admin UI](../setup-guide/platform-admin-ui.html), which is the replacement for the legacy admin UI.Both UIs are available as separate downloads from the [Backstage download site](https://backstage.forgerock.com/downloads):- To install the Platform admin UI, follow the steps in [Install the Platform admin UI for standalone IDM](../setup-guide/platform-admin-ui.html). - To continue using the legacy admin UI, follow the steps in [Install the legacy admin UI](../setup-guide/legacy-admin-ui.html). | The first time you log in, use the default administrative credentials, (Login: openidm-admin, Password: openidm-admin). You should now see the Dashboard screen, with *quick start cards* for common administrative tasks. 1. Reconcile the two resources as follows: Click Configure > Mappings, select the `systemCsvfileAccounts_managedUser` mapping, and click Reconcile. 2. After reconciliation, display the user records in both the source and target resources. Select the Association tab and scroll down to the bottom of the page to see the resulting source and target users.