Multiple CSV Cloud connector

The Multiple CSV Cloud connector allows for resources such as users and groups to be retrieved from one or more CSV files, as defined by the connector configuration, across different cloud storage platforms.

This connector does not verify CSV data. You must ensure that your CSV file is complete and properly formed before using the connector.
Do not remove or replace CSV files that are the source or target of an active scheduled reconciliation.
Modifying the CSV header by adding, removing, or modifying the columns requires reconfiguring the connector for the changes to take effect.

Install the Multiple CSV Cloud connector

To check for an Advanced Identity Cloud application for this connector, refer to:

You can download any connector from Backstage, but some are included in the default deployment for Advanced Identity Cloud, IDM, or RCS. When using an included connector, you can skip installing it and move directly to configuration.

Connector included in default deployment
Connector	IDM	RCS
Multiple CSV Cloud	No	No

Download the connector .jar file from Backstage.

If you are running the connector locally, place it in the /path/to/openidm/connectors directory, for example:
```
mv ~/Downloads/multicsvfilecloud-connector-1.5.20.31.jar /path/to/openidm/connectors/
```
If you are using a remote connector server (RCS), place it in the /path/to/openicf/connectors directory on the RCS.

Configure the Multiple CSV Cloud connector

Create a connector configuration using the IDM admin UI:

From the navigation bar, click Configure > Connectors.
On the Connectors page, click New Connector.
On the New Connector page, type a Connector Name.
From the Connector Type drop-down list, select Multiple CSV Cloud Connector - 1.5.20.31.
Complete the Base Connector Details.

For a list of all configuration properties, refer to Multiple CSV Cloud Connector Configuration
Click Save.

When your connector is configured correctly, the connector displays as Active in the admin UI.

Refer to this procedure to create a connector configuration over REST.

{
  "configurationProperties": {
    "minBytesToEstimate": 100000000,
    "multiValuedFieldDelimiter": ";",
    "newLineString": "\n",
    "quotationMode": "ALL",
    "csvFiles": [
      {
        "objectType": "__ACCOUNT__",
        "path": "_CHANGEME_",
        "name": "_CHANGEME_",
        "uid": "_CHANGEME_"
      }
    ],
    "quoteCharacter": "\"",
    "escapeCharacter": "\\",
    "fieldDelimiter": ",",
    "ignoreSurroundingSpaces": false,
    "clientId": "_CHANGEME_",
    "clientSecret": "_CHANGEME_",
    "jwtPem": "_CHANGEME_",
    "jwtIssuer": "_CHANGEME_",
    "storageType": "_CHANGEME_"
  }
}

minBytesToEstimate

The total number of rows in the CSV file will begin to be estimated when the file size in bytes exceeds the value specified in this field. The default value is 100MB.

multiValuedFieldDelimiter

String value that delimits each field inside a multivalued field.

newLineString

Indicates the character that represents a new line in the CSV file.

quotationMode

Defines quoting behavior.

Available values:
- ALL: Quotes all fields.
- ALL_NON_NULL: Quotes all non-null fields.
- MINIMAL: Quotes fields that contain special characters such as a field delimiter, quote character, or any of the characters in the line separator string.
- NON_NUMERIC: Quotes all non-numeric fields.
- NONE: Never quotes fields.

csvFiles

This field is required. It’s an array of objects, where each object represents the CSV file associated with a corresponding object type.

objectType

The name of the object type.

path

The path where the CSV file is located.

name

The field that represents the name of the record.

uid

The field that represents the ID of the record.

quoteCharacter

Defines the characters used to enclose values.

escapeCharacter

Defines the characters used to escape special characters.

fieldDelimiter

String value that delimits each field. Defaults to ",".

ignoreSurroundingSpaces

Ignores whitespace around field separators. Can alter CSV structure. Defaults to false.

clientId

The client identifier for OAuth. Only required for AWS.

clientSecret

Secure client secret for OAuth. Only required for AWS.

jwtPem

It is a String. Refers to the cryptographic keys used to sign and verify jwts, presented in PEM format. Only required for Google.

jwtIssuer

The entity that issues the token. Only required for Google.

storageType

It is a String and its value can be:

Google
Azure
AWS

The character used for quoteCharacter, escapeCharacter, and fieldDelimiter must be unique.

Multiple CSV Cloud remote connector

If you want to run this connector outside of PingOne Advanced Identity Cloud or IDM, you can configure the Multiple CSV Cloud connector as a remote connector. Java Connectors installed remotely on a Java Connector Server function identically to those bundled locally within PingOne Advanced Identity Cloud or installed locally on IDM.

You can download the Multiple CSV Cloud connector from here.

Refer to Remote connectors for configuring the Multiple CSV Cloud remote connector.

Configure connection pooling

The Multiple CSV Cloud connector uses a non-poolable mechanism to manage connections. Learn more about the different pooling mechanisms in Connectors by pooling mechanism.

Use the Multiple CSV Cloud connector

You can use the Multiple CSV Cloud connector to perform the following actions.

List all Multiple CSV Cloud accounts

This example queries all Multiple CSV Cloud accounts:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--request GET \
"http://localhost:8080/openidm/system/multicsvcloud/__ACCOUNT__?_queryFilter=true"
{
  "result": [
    {
      "_id": "106879",
      "User_Last_Name": "Abel",
      "Groups": "ResponderProject",
      "__NAME__": "106879",
      "User_First_Name": "Marcela"
    },
    {
      "_id": "615MAB77",
      "User_Last_Name": "Broome",
      "Groups": "Supervisor",
      "__NAME__": "615MAB77",
      "User_First_Name": "Melissa"
    },
    ...
    {
      "_id": "127403",
      "User_Last_Name": "Stevens",
      "Groups": "Supervisor",
      "__NAME__": "127403",
      "User_First_Name": "Ginelys"
    },
  ],
  "resultCount": 10,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

The value of the totalPagedResults field is estimated for files larger than 100000000 (100MB).

The Multiple CSV Cloud Connector supports different types of filters like:

equals : eq
contains : co
startswith : sw
endwith : ew
lessoreq : le
lessthan : lt
greateroreq : ge
greaterthan : gt
not : !

Get Multiple CSV Cloud user account by ID

This example queries Multiple CSV Cloud user account by ID:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--request GET \
"http://localhost:8080/openidm/system/multicsvcloud/__ACCOUNT__/106879"
{
  "_id": "106879",
  "User_Last_Name": "Abel",
  "Groups": "ResponderProject",
  "__NAME__": "106879",
  "User_First_Name": "Marcela"
}

List all Multiple CSV Cloud user accounts by IDs

This example queries all Multiple CSV Cloud users by their IDs:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--request GET \
"http://localhost:8080/openidm/system/multicsvcloud/__ACCOUNT__?_queryId=query-all-ids"
{
  "result": [
    {
      "_id": "106879"
    },
    {
      "_id": "615MAB77"
    },
    ...
    {
      "_id": "127403"
    }
  ],
  "resultCount": 1234,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

OpenICF Interfaces Implemented by the Multi CSV Cloud Connector

The Multi CSV Cloud Connector implements the following OpenICF interfaces. For additional details, see ICF interfaces:

Authenticate

Provides simple authentication with two parameters, presumed to be a user name and password.

Create

Creates an object and its uid.

Delete

Deletes an object, referenced by its uid.

Resolve Username

Resolves an object by its username and returns the uid of the object.

Schema

Describes the object types, operations, and options that the connector supports.

Script on Connector

Enables an application to run a script in the context of the connector.

Any script that runs on the connector has the following characteristics:

The script runs in the same execution environment as the connector and has access to all the classes to which the connector has access.
The script has access to a connector variable that is equivalent to an initialized instance of the connector. At a minimum, the script can access the connector configuration.
The script has access to any script arguments passed in by the application.

Search

Searches the target resource for all objects that match the specified object class and filter.

Test

Tests the connector configuration.

Testing a configuration checks all elements of the environment that are referred to by the configuration are available. For example, the connector might make a physical connection to a host that is specified in the configuration to verify that it exists and that the credentials that are specified in the configuration are valid.

This operation might need to connect to a resource, and, as such, might take some time. Do not invoke this operation too often, such as before every provisioning operation. The test operation is not intended to check that the connector is alive (that is, that its physical connection to the resource has not timed out).

You can invoke the test operation before a connector configuration has been validated.

Update

Updates (modifies or replaces) objects on a target resource.

Multi CSV Cloud Connector Configuration

The Multi CSV Cloud Connector has the following configurable properties:

Configuration properties

Property Type Default Encrypted⁽¹⁾ Required⁽²⁾

Property	Type	Default	Required⁽²⁾
`csvFiles`	`Map[]`	`['{path=null, uid=null, password=null, multiValuedAttr=[], name=null, objectType=null}']`	Yes
The full path to the CSV file that is the data source for this connector.
`quoteCharacter`	`String`	`"`	No
The character in the CSV file that is used to encapsulate strings. The default value is double quote (")
`fieldDelimiter`	`String`	`,`	No
The character in the CSV file that is used to separate field values. The default value is a comma (,)
`escapeCharacter`	`String`	`\`	No
The character in the CSV file that is used to escape characters. The default value is a backslash ("\")
`newLineString`	`String`	`\n`	No
The character string in the CSV file that is used to terminate each line. The default value is a newline character(" ")
`spaceReplacementString`	`String`	`_`	No
The character(s) used to replace spaces within column names.The default value is an underscore (_)
`minBytesToEstimate`	`double`	`1.0E8`	No
Minimum number of bytes from which the number of records will be estimated. The default value is 1E+8 bytes
`multiValuedFieldDelimiter`	`String`	`null`	No
The character in the CSV file that is used to separate Multi valued field values. The default value is a semicolon (;)
`quotationMode`	`String`	`ALL`	No
Specifies the quotation behavior. For instance, ALL applies quotation to all columns. The default value is "ALL"
`ignoreSurroundingSpaces`	`boolean`	`false`	No
True if spaces around values are ignored, false if they are treated as part of the value. The default value is "false"

csvFiles

Map[]

['{path=null, uid=null, password=null, multiValuedAttr=[], name=null, objectType=null}']

Yes

The full path to the CSV file that is the data source for this connector.

quoteCharacter

String

"

The character in the CSV file that is used to encapsulate strings. The default value is double quote (")

fieldDelimiter

String

,

The character in the CSV file that is used to separate field values. The default value is a comma (,)

escapeCharacter

String

\

The character in the CSV file that is used to escape characters. The default value is a backslash ("\")

newLineString

String

\n

The character string in the CSV file that is used to terminate each line. The default value is a newline character(" ")

spaceReplacementString

String

_

The character(s) used to replace spaces within column names.The default value is an underscore (_)

minBytesToEstimate

double

1.0E8

Minimum number of bytes from which the number of records will be estimated. The default value is 1E+8 bytes

multiValuedFieldDelimiter

String

null

The character in the CSV file that is used to separate Multi valued field values. The default value is a semicolon (;)

quotationMode

String

ALL

Specifies the quotation behavior. For instance, ALL applies quotation to all columns. The default value is "ALL"

ignoreSurroundingSpaces

boolean

false

True if spaces around values are ignored, false if they are treated as part of the value. The default value is "false"

⁽¹⁾ Whether the property value is considered confidential, and is therefore encrypted in IDM.

⁽²⁾ A list of operations in this column indicates that the property is required for those operations.

Basic Configuration Properties

Property Type Default Encrypted⁽¹⁾ Required⁽²⁾

Property	Type	Default	Encrypted⁽¹⁾	Required⁽²⁾
`serviceUri`	`String`	`null`		Yes
The service endpoint URI.
`tokenEndpoint`	`String`	`null`		No
When using OAUTH as authentication method, this property defines the endpoint where a new access token should be queried for (https://myserver.com/oauth2/token).
`clientId`	`String`	`null`		Yes
The client identifier for OAuth2.
`clientSecret`	`GuardedString`	`null`	Yes	No
Secure client secret for OAuth2.
`scope`	`String`	`null`		No
The OAuth2 scope to use.
`jwtPem`	`String`	`null`		No
The contents of the private key of the PEM file
`storageType`	`String`	`null`		No
The type of storage to use for the CSV files. Options are Google, AWS and Azure
`jwtIssuer`	`String`	`null`		No
The issuer of the JWT token.

serviceUri

String

null

Yes

The service endpoint URI.

tokenEndpoint

String

null

When using OAUTH as authentication method, this property defines the endpoint where a new access token should be queried for (https://myserver.com/oauth2/token).

clientId

String

null

Yes

The client identifier for OAuth2.

clientSecret

GuardedString

null

Yes

Secure client secret for OAuth2.

scope

String

null

The OAuth2 scope to use.

jwtPem

String

null

The contents of the private key of the PEM file

storageType

String

null

The type of storage to use for the CSV files. Options are Google, AWS and Azure

jwtIssuer

String

null

The issuer of the JWT token.

⁽¹⁾ Whether the property value is considered confidential, and is therefore encrypted in IDM.

⁽²⁾ A list of operations in this column indicates that the property is required for those operations.

ICF 1.5.20.31

Multiple CSV Cloud connector

Install the Multiple CSV Cloud connector

Configure the Multiple CSV Cloud connector

Multiple CSV Cloud remote connector

Configure connection pooling

Use the Multiple CSV Cloud connector

List all Multiple CSV Cloud accounts

Get Multiple CSV Cloud user account by ID

List all Multiple CSV Cloud user accounts by IDs

OpenICF Interfaces Implemented by the Multi CSV Cloud Connector

Multi CSV Cloud Connector Configuration

Configuration properties

Basic Configuration Properties