Synchronize data from a CSV file to IDM

This sample demonstrates one-way synchronization from an external resource to an IDM repository.

The external resource in this case is a simple CSV file. User objects in that file are synchronized with the managed users in the IDM repository.

Sample overview

IDM connects data objects held in separate resources by mapping one object to another. To connect to external resources, IDM uses connectors, that are configured for each external resource.

When objects in one external resource change, IDM determines how the changes affect the objects in the connected resource, and can make the changes in that resource as necessary. This sample demonstrates how IDM does this by using reconciliation. Reconciliation compares the objects in one resource to the mapped objects in another resource. For a complete explanation of reconciliation and synchronization, see Synchronization types.

In this sample, IDM connects to a CSV file that holds sample user data. The CSV file is configured as the authoritative source. A mapping is configured between objects in the CSV file and managed user objects in the IDM repository.

Note that you can use IDM to synchronized objects between two external resources without going through the IDM repository. In such a case, objects are synchronized directly through connectors to the external resources.

This sample involves only one external resource. In practice, you can connect as many resources as needed for your deployment.

Sample configuration files

The configuration files for this sample are located in the /path/to/openidm/samples/sync-with-csv/conf directory. When you start IDM with the -p project variable (./startup.sh -p samples/sync-with-csv), the project location (&{idm.instance.dir}) is set to a value of samples/sync-with-csv. All subsequent paths use this project location as a base. Throughout this documentation, you will see things like "…in your project’s conf/ directory…". The "project" refers to the value of the &{idm.instance.dir} variable.

The following configuration files play important roles in this sample:

samples/sync-with-csv/conf/provisioner.openicf-csvfile.json

This file provides the configuration for this instance of the CSV connector. It describes, among other things, the connector version, the location of the CSV file resource, and the object types that are supported for this connection. For a complete understanding of connector configuration files, see Configure connectors.

samples/sync-with-csv/conf/sync.json

This file, also called a mapping file, defines the configuration for reconciliation and synchronization. This sample file includes only one mapping - systemCsvfileAccounts_managedUser. The mapping specifies the synchronization configuration between the CSV file (source) and the IDM repository (target). Examine the file to see how objects are mapped between the two resources, and the actions that IDM should take when it finds objects in specific situations:

{
    "mappings": [
        {
            "name" : "systemCsvfileAccounts_managedUser",
            "source" : "system/csvfile/account",
            "target": "managed/user",
            "correlationQuery": {
                "type": "text/javascript",
                "source": "var query = {'_queryId' : 'for-userName',
                    'uid' :  source.name};query;"
            },
            "properties": [
                {
                    "source": "email",
                    "target": "mail"
                },
                {
                    "source": "firstname",
                    "target": "givenName"
                },
                {
                    "source": "lastname",
                    "target": "sn"
                },
                {
                    "source": "description",
                    "target": "description"
                },
                {
                    "source": "_id",
                    "target": "_id"
                },
                {
                    "source": "name",
                    "target": "userName"
                },
                {
                    "source": "password",
                    "target": "password"
                },
                {
                    "source" : "mobileTelephoneNumber",
                    "target" : "telephoneNumber"
                },
                {
                    "source" : "roles",
                    "transform" : {
                        "type" : "text/javascript",
                        "source" : "var _ = require('lib/lodash'); _.map(source.split(','),
                         function(role) { return {'_ref': 'internal/role/' + role} });"
                    },
                    "target" : "authzRoles"
                }
            ],
            "policies": [
                {
                    "situation": "CONFIRMED",
                    "action": "UPDATE"
                },
                {
                    "situation": "FOUND",
                    "action": "IGNORE"
                },
                {
                    "situation": "ABSENT",
                    "action": "CREATE"
                },
                {
                    "situation": "AMBIGUOUS",
                    "action": "IGNORE"
                },
                {
                    "situation": "MISSING",
                    "action": "IGNORE"
                },
                {
                    "situation": "SOURCE_MISSING",
                    "action": "IGNORE"
                },
                {
                    "situation": "UNQUALIFIED",
                    "action": "IGNORE"
                },
                {
                    "situation": "UNASSIGNED",
                    "action": "IGNORE"
                }
            ]
        }
    ]
}

Source and target paths that start with managed, such as managed/user, always refer to objects in the IDM repository. Paths that start with system, such as system/csvfile/account, refer to external objects, in this case, objects in the CSV file.

When you start a reconciliation, IDM queries all users in the source, and then creates, deletes, or modifies users in the IDM repository, as mapped in conf/sync.json.

For more information about synchronization, reconciliation, and mappings, see Synchronization.

samples/sync-with-csv/conf/schedule-reconcile_systemCsvAccounts_managedUser.json

The sample schedule configuration file defines a task that launches a reconciliation every minute for the mapping named systemCsvfileAccounts_managedUser. The schedule is disabled by default:

{
    "enabled" : false,
    "type": "simple",
    "repeatInterval": 3600000,
    "persisted" : true,
    "concurrentExecution" : false,
    "misfirePolicy" : "fireAndProceed",
    "invokeService" : "sync",
    "invokeContext" : {
        "action" : "reconcile",
        "mapping" : "systemCsvfileAccounts_managedUser"
    }
}

IDM regularly scans the conf/ directory for any schedule configuration files.

Apart from the scheduled reconciliation run, you can also start reconciliation run through the REST interface. The call to the REST interface is an HTTP POST such as the following:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request POST \
"http://localhost:8080/openidm/recon?_action=recon&mapping=systemCsvfileAccounts_managedUser&waitForCompletion=true"

The waitForCompletion=true parameter specifies that the operation should return only when it has completed.

samples/sync-with-csv/data/csvConnectorData.csv

This CSV file is the external resource or data store in this sample. The file contains two users, bjensen and scarter. During the sample, you will reconcile those users from the CSV file to the managed user repository.

Run the sample

To run this sample, start IDM with the configuration for the sample:

/path/to/openidm/startup.sh -p samples/sync-with-csv

You can work through the sample using the command line, or using the admin UI:

Use the Command Line

When you have started IDM, reconcile the objects in both resources.

You can trigger the reconciliation either by setting "enabled" : true in the schedule configuration file (conf/schedule-reconcile_systemCsvAccounts_managedUser.json) and then waiting until the scheduled reconciliation happens, or by running the following curl command:
```
curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request POST \
"http://localhost:8080/openidm/recon?_action=recon&mapping=systemCsvfileAccounts_managedUser&waitForCompletion=true"
```
Successful reconciliation returns a reconciliation run ID, and the status of the reconciliation operation, as follows:
```
{
  "_id":"2d87c817-3d00-4776-a705-7de2c65937d8",
  "state":"SUCCESS"
}
```

Display the managed user records that were created by the reconciliation operation.

You can use any REST client to query the repository. Perform an HTTP GET on the URL "http://localhost:8080/openidm/managed/user?_queryFilter=true" with the headers "X-OpenIDM-Username: openidm-admin" and "X-OpenIDM-Password: openidm-admin". The following example uses the curl command to get all managed user records, in JSON format:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/managed/user?_queryFilter=true"
{
  "result": [
    {
      "_id": "bjensen",
      "_rev": "00000000e17186b6",
      "mail": "bjensen@example.com",
      "givenName": "Barbara",
      "sn": "Jensen",
      "description": "Created By CSV",
      "userName": "bjensen",
      "telephoneNumber": "1234567",
      "accountStatus": "active",
      "effectiveAssignments": [],
      "effectiveRoles": []
    },
    {
      "_id": "scarter",
      "_rev": "00000000970685c3",
      "mail": "scarter@example.com",
      "givenName": "Steven",
      "sn": "Carter",
      "description": "Created By CSV",
      "userName": "scarter",
      "telephoneNumber": "1234567",
      "accountStatus": "active",
      "effectiveAssignments": [],
      "effectiveRoles": []
    }
  ],
  ...
}

You can user any query filter to return the information you need. For more information, see Define and call data queries.

Now display user bjensen’s record by appending her user ID to the URL:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/managed/user/bjensen"
{
  "_id": "bjensen",
  "_rev": "00000000e17186b6",
  "mail": "bjensen@example.com",
  "givenName": "Barbara",
  "sn": "Jensen",
  "description": "Created By CSV",
  "userName": "bjensen",
  "telephoneNumber": "1234567",
  "accountStatus": "active",
  "effectiveAssignments": [],
  "effectiveRoles": []
}

This command returns bjensen’s complete user record.

Restrict the query output with the fields parameter, as follows:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/managed/user?_queryFilter=true&_fields=userName,mail"
{
  "result": [
    {
      "_id": "bjensen",
      "_rev": "00000000e17186b6",
      "mail": "bjensen@example.com",
      "userName": "bjensen"
    },
    {
      "_id": "scarter",
      "_rev": "00000000970685c3",
      "mail": "scarter@example.com",
      "userName": "scarter"
    }
  ],
  ...
}

To test the scheduled reconciliation, add a user to the CSV data file, samples/sync-with-csv/data/csvConnectorData.csv . For example, add user jberg as follows:

"description", "uid", "username", "firstname", "lastname", "email", "mobile...
"Created ...", "bjensen", "bjensen@example.com", "Barbara", "Jensen", "bjensen@example.com", "123456...
"Created ...", "scarter", "scarter@example.com", "Steven", "Carter", "scarter@example.com", "123456...
"Created ...", "jberg", "jberg@example.com", "James", "Berg", "jberg@example.com", "123456...

If you enabled the scheduled reconciliation in Step 1, you can simply wait for the reconciliation operation to run. Otherwise, run the reconciliation manually with the same command you used in that step.

After the reconciliation has run, query the managed user repository to see the new user in the list of managed users:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/managed/user?_queryFilter=true&_fields=_id"
{
  "result": [
    {
      "_id": "bjensen",
      "_rev": "00000000e17186b6"
    },
    {
      "_id": "scarter",
      "_rev": "00000000970685c3"
    },
    {
      "_id": "jberg",
      "_rev": "00000000ea628233"
    }
  ],
  ...
}

To view the reconciliation details, query the reconciliation using its id:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/recon/assoc/2d87c817-3d00-4776-a705-7de2c65937d8"
{
  "_id": "2d87c817-3d00-4776-a705-7de2c65937d8",
  "_rev": "1",
  "mapping": "systemCsvfileAccounts_managedUser",
  "sourceResourceCollection": "managed/user",
  "targetResourceCollection": "system/csv/account",
  "isAnalysis": "false",
  "finishTime": "2022-05-01T23:36:24.434153Z"
}

For more information on reconciliation operations via REST, see Manage reconciliation.

You configure the action that IDM takes for each situation in the mapping file, conf/sync.json. For the list of all possible situations and actions, see Synchronization situations and actions.

If you’ve enabled audit logging, you can view the reconciliation details in the openidm/audit/recon.audit.json file.

Use the admin UI

IDM includes a browser-based Administrative User Interface, known as the admin UI.

After starting IDM, access the admin UI by navigating to https://localhost:8443/admin. The first time you log in, use the default administrative credentials, (Login: openidm-admin, Password: openidm-admin).

You should now see the Dashboard screen, with quick start cards for common administrative tasks.

Reconcile the two resources as follows:

Click Configure > Mappings, select the systemCsvfileAccounts_managedUser mapping, and click Reconcile.
After reconciliation, display the user records in both the source and target resources.

Select the Association tab and scroll down to the bottom of the page to see the resulting source and target users.

IDM 7.2.2

Synchronize data from a CSV file to IDM

Sample overview

Sample configuration files

Run the sample