--- title: Migrate to file-based configuration description: Prior to AM 8.0, all information (configuration and data) was stored in LDAP directories. AM 8.1 can read its configuration from JSON files instead of directory servers. component: pingam version: 8.1 page_id: pingam:upgrade:migrate-to-fbc canonical_url: https://docs.pingidentity.com/pingam/8.1/upgrade/migrate-to-fbc.html keywords: ["Migrate", "FBC", "File-based configuration"] page_aliases: ["upgrade-guide:migrate-to-fbc.adoc"] section_ids: separate-config: Separate dynamic data from configuration data export-existing-config: Export existing configuration data migrate-config: Migrate exported configuration to FBC copy-fbc-to-am: Provide configuration files to AM start-am-fbc: Start AM in FBC mode migrate-fbc-troubleshooting: Troubleshoot your FBC migration --- # Migrate to file-based configuration Prior to AM 8.0, all information (configuration *and* data) was stored in LDAP directories. AM 8.1 can read its configuration from JSON files instead of directory servers. File-based configuration (FBC) is best-suited to a DevOps-style deployment, with the associated tools and practices of that approach. Static FBC data is written to configuration files in the file system and checked into a source control system, such as Git. AM instances are created as Docker images, with the FBC incorporated into the image. ![Kubernetes deployment using file-based configuration.](../_images/docker-deployment.png) You can insert variables into these configuration files before you check them into source control. The variables are substituted with the appropriate values at runtime when you start the Docker container. Using variables lets you reuse the same base configuration files for multiple instances, and different staging environments. For example, development, QA, or pre-production, which are then promoted to production. In an existing deployment, you can migrate your configuration to FBC *after you've upgraded* the configuration to AM 8.1. | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Before you start, read [How AM reads FBC](../installation/fbc.html#how-am-reads-FBC) to understand the file-based configuration layers and their order of precedence. | These topics assume you've already upgraded to AM 8.1 using the [upgrade wizard](upgrade-servers.html#upgrade-wizard). If you upgraded using, [amupgrade and Amster](upgrade-servers.html#upgrade-amupgrade), first deploy AM 8.1 and import the upgraded `amster` configuration. Then continue with the migration to FBC. Migrating to FBC involves the following high-level steps: 1. [Separate dynamic data from configuration data](#separate-config) 2. [Export existing configuration data](#export-existing-config) 3. [Migrate exported configuration to FBC](#migrate-config) 4. [Provide configuration files to AM](#copy-fbc-to-am) 5. [Start AM in FBC mode](#start-am-fbc) ## Separate dynamic data from configuration data Most deployments have one or more LDAP directories to store static configuration *and* data (identities, CTS tokens, policies, agents, and applications). If you store configuration and data in a single LDAP directory, you must separate the dynamic data first. This section assumes you have stored identities in a separate datastore and focuses on CTS tokens, policies, agents, and applications. Export the data from your existing DS server and import it into one or more DS servers specifically for that data. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | | You can use your existing directory to store your policies and applications, to avoid the overhead of setting up and managing additional DS instances during the migration. In this case, overwrite the contents of the directory with only the dynamic data. If you do use the existing instance, backup directory so that you can restore it if issues occur during the migration. | To assist with the export, this table shows the default backend IDs and base DNs for AM data stored in DS directories. The table assumes you installed DS with the appropriate setup profiles. Adjust the values to reflect how you configured your DS server to store this data. | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | There is no DS setup profile for UMA-related data. This table therefore shows the values suggested in [Prepare external UMA datastores](../uma/prepare-uma-store.html). | | Data type | Backend ID | Base DN | | --------------------------------- | ------------ | ------------------------------------------- | | CTS tokens | `amCts` | `ou=famrecords,ou=openam-session,ou=tokens` | | Policies, applications and agents | `cfgStore` | `ou=services,ou=am-config` | | UMA-related data | `umaRsStore` | `dc=uma-resources,dc=example,dc=com` | 1. Set up at least one external PingDS instance for your exported data, following these instructions: * [Prepare policy and application stores](../setup/prepare-policy-and-application-store.html) * [Configure CTS token stores](../cts/cts-openam-config.html) * [Prepare external UMA datastores](../uma/prepare-uma-store.html) 2. Use the `export-ldif` command on your current DS server instance to export the policies, applications, agents, CTS tokens and any UMA-related data from your existing configuration store. * This command exports the CTS tokens to an LDIF file named `output-cts.ldif`: ```bash $ /path/to/opendj/bin/export-ldif \ --hostname ds.example.com \ --port 1636 \ --useSsl \ --usePkcs12TrustStore ~/path/to/opendj/config/keystore \ --trustStorePassword:file ~/path/to/opendj/config/keystore.pin \ --bindDn uid=admin \ --bindPassword str0ngAdm1nPa55word \ --backendId amCts \ --ldifFile output-cts.ldif Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 scheduled to start immediately …​ Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 has been successfully completed ``` * This command exports UMA-related data to an LDIF file named `output-uma.ldif`: ```bash $ /path/to/opendj/bin/export-ldif \ --hostname ds.example.com \ --port 1636 \ --useSsl \ --usePkcs12TrustStore ~/path/to/opendj/config/keystore \ --trustStorePassword:file ~/path/to/opendj/config/keystore.pin \ --bindDn uid=admin \ --bindPassword str0ngAdm1nPa55word \ --backendId umaRsStore \ --ldifFile output-uma.ldif Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 scheduled to start immediately …​ Export task ExportTask-16a9dce5-ceb1-41f0-a94d-154246ba7393 has been successfully completed ``` * Follow the steps in [Migrate policy or application data to DS](../setup/prepare-policy-and-application-store.html#migrate-policy-and-application-data) to export the policy and application data. You can find more information about this command in [export-ldif](https://docs.pingidentity.com/pingds/8.1/tools-reference/export-ldif.html). 3. Import the dynamic data into the new DS server or servers you've installed for this purpose. * This command imports the CTS tokens into a DS server instance at `ds-cts.example.com`: ```bash $ /path/to/opendj/bin/import-ldif \ --hostname ds-cts.example.com \ --port 1636 \ --useSsl \ --usePkcs12TrustStore ~/path/to/opendj/config/keystore \ --trustStorePassword:file ~/path/to/opendj/config/keystore.pin \ --bindDn uid=admin \ --bindPassword str0ngAdm1nPa55word \ --backendId amCts \ --ldifFile output-cts.ldif Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b scheduled to start immediately …​ Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b has been successfully completed ``` * This command imports the UMA-related data into a DS server instance at `ds-uma.example.com`: ```bash $ /path/to/opendj/bin/import-ldif \ --hostname ds-uma.example.com \ --port 1636 \ --useSsl \ --usePkcs12TrustStore ~/path/to/opendj/config/keystore \ --trustStorePassword:file ~/path/to/opendj/config/keystore.pin \ --bindDn uid=admin \ --bindPassword str0ngAdm1nPa55word \ --backendId umaRsStore \ --ldifFile output-uma.ldif Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b scheduled to start immediately …​ Import task ImportTask-2f93d32f-0599-4215-8d3e-ecb77164d64b has been successfully completed ``` * Follow the steps in [Migrate policy or application data to DS](../setup/prepare-policy-and-application-store.html#migrate-policy-and-application-data) to import the policy and application data to the DS server you've set up for that purpose. You can find more information about this command in [import-ldif](https://docs.pingidentity.com/pingds/8.1/tools-reference/import-ldif.html). 4. On your AM configuration server, delete the dynamic data you've exported. | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Although you don't *have* to do this, it does result in a cleaner migration of static configuration. You must have a clear understanding of how your data is stored in the underlying LDAP directory before you delete the dynamic data you don't want to store in FBC. | To delete all policy definitions and applications: 1. Create a text file that includes the following subtrees: ```bash $ cat subtrees-to-delete.txt ``` Output ```bash o=sunamhiddenrealmdelegationservicepermissions,ou=services,ou=am-config ou=AgentService,ou=services,ou=am-config ou=sunEntitlementService,ou=services,ou=am-config ou=sunEntitlementIndexes,ou=services,ou=am-config ou=sunFMCOTConfigService,ou=services,ou=am-config ou=sunFMIDFFMetadataService,ou=services,ou=am-config ou=sunFMSAML2MetadataService,ou=services,ou=am-config ou=sunFMWSFederationMetadataService,ou=services,ou=am-config ``` 2. Send this list to the `ldapdelete` command on standard input: ```bash $ cat subtrees-to-delete.txt | /path/to/opendj/bin/ldapdelete \ --hostname ds.example.com \ --port 1636 \ --useSsl \ --usePkcs12TrustStore ~/path/to/opendj/config/keystore \ --trustStorePassword:file ~/path/to/opendj/config/keystore.pin \ --bindDN "uid=admin" \ --bindPassword str0ngAdm1nPa55word \ --deleteSubtree # DELETE operation successful for DN o=sunamhiddenrealmdelegationservicepermissions,ou=services,ou=am-config …​ ``` ## Export existing configuration data The migration utility consumes an LDIF export of the existing configuration. Use the `export-ldif` command to export the data, supplying the following parameters: * The backend ID of the configuration store, `cfgStore` by default. * The base DN of the branch containing the configuration, `ou=am-config` by default. * An LDIF file to store the exported configuration. The following example, run on the DS server, assumes the default parameters when DS is [installed for AM configuration](https://docs.pingidentity.com/pingds/8.1/install-guide/profile-am-config.html): ```bash $ /path/to/opendj/bin/export-ldif \ --backendId cfgStore \ --includeBranch ou=am-config \ --ldifFile /path/to/export-config/exported-configuration.ldif \ --offline …​ category=BACKEND severity=INFO seq=34 msg=Exported 1528 entries and skipped 0 in 0 seconds (average rate 24254.0/sec) ``` The exported file contains the existing configuration under `ou=am-config`: ```bash $ more /path/to/export-config/exported-configuration.ldif dn: ou=am-config objectClass: top objectClass: untypedObject ou: am-config …​ ``` ## Migrate exported configuration to FBC The migration utility is an enhanced version of the AM `amupgrade` command, available in the AM `.zip` file. 1. If you don't already have it, download `AM-8.1.0.zip` from the [Ping Identity Download Center](https://backstage.pingidentity.com/downloads). 2. Extract the contents of the `.zip` archive. 3. In the extracted directory, locate and extract the `Config-Upgrader-8.1.0.zip` archive. 4. To migrate the LDAP configuration, run the `amupgrade` command with the `migrationMode` argument: ```bash $ /path/to/amupgrade \ --migrationMode \ --inputConfig input-directory \ --output output-directory \ /path/to/amupgrade/rules/fbc/migrate-to-fbc.groovy ``` > **Collapse: command-line arguments** > > * `--amsterVersion, -a` > > The version of Amster that will be used to import the configuration (for example 8.0.0). This parameter is required unless you provide the `--migrationMode` or `--fileBasedMode` arguments. > > * `--baseDn, -d` > > Lets you set a custom base DN. > > * `--clean` > > The migration should remove all existing files from the output location directory. > > Default: true > > * `--fileBasedMode, -m` > > The configuration to be upgraded is file-based configuration (not Amster-exported configuration). > > Default: false > > * `--ignoreNoRuleTracking` > > Don't check for the existence of the Configuration Version Service. This lets you run the migration tool against individual files. However, it means that rules can be run multiple times against those files. Most rules don't support this. Use with caution. > > Default: false > > * `--ignoreRequiredSecretMapping` > > Prevents the upgrade from terminating due to missing required secret label mappings. Use with caution. > > Default: false > > * `-i, --inputConfig` > > The directory containing the LDIF configuration files that you're migrating. > > * `--log-output, -l` > > The logging output file; logs at FINE by default and FINEST in verbose mode. > > * `--migration-mode` > > The upgrade command should *migrate* the configuration from LDIF data to file-based (JSON) data. > > * `--prettyArrays` > > Whether to print each JSON array value on a new line. > > Default: false > > * `-o | --output directory` > > The directory on the file system to which the migration utility writes the generated FBC. > > * `--secretsConfigFile, -f` > > The properties file that contains settings necessary to migrate credentials to secrets. This file is required if your rules contain secrets upgrades. Read the `sampleconfig.properties` file for more details. > > * `--sectionsSource, -s` > > A file that specifies the structure (*section* details) of the output configuration. > > Without this option, the migration utility generates the configuration properties as flat files. > > This file can be a `section.properties` file for a specific service or an `AM.war` file that includes the structure for all services. > > > **Collapse: Example file for JSON audit handler** > > > > ```properties > > ######################################################################################################################## > > # Common handler section properties > > ######################################################################################################################## > > commonHandler=enabled > > commonHandler=topics > > > > ######################################################################################################################## > > # Common handler plugin section properties > > ######################################################################################################################## > > commonHandlerPlugin=handlerFactory > > > > ######################################################################################################################## > > # JSON handler section properties > > ######################################################################################################################## > > jsonConfig=location > > jsonConfig=elasticsearchCompatible > > jsonConfig=rotationRetentionCheckInterval > > jsonFileRotation=rotationEnabled > > jsonFileRotation=rotationMaxFileSize > > jsonFileRotation=rotationFilePrefix > > jsonFileRotation=rotationFileSuffix > > jsonFileRotation=rotationInterval > > jsonFileRotation=rotationTimes > > jsonFileRetention=retentionMaxNumberOfHistoryFiles > > jsonFileRetention=retentionMaxDiskSpaceToUse > > jsonFileRetention=retentionMinFreeSpaceRequired > > jsonBuffering=bufferingMaxSize > > jsonBuffering=bufferingWriteInterval > > ``` > > * `rules-file-path` > > The path to the upgrade rules file. > > Although you're not using this command to upgrade to a new version, the `amupgrade` command requires an upgrade rules file. > > Use the `noop.groovy` file (located in (`/path/to/amupgrade/rules/fbc/noop.groovy`) to migrate the configuration without applying any rules. > > * `--version, -V` > > Prints the version of this configuration upgrader tool and exits. > > * `--verbose, -v` > > Logs verbose output; outputs at FINE level to the console and FINEST level to files. For example: ```bash $ /path/to/amupgrade \ --migrationMode \ --inputConfig /path/to/export-config/ \ --output /path/to/fbc/config/ \ /path/to/amupgrade/rules/fbc/migrate-to-fbc.groovy ``` In this example, the new JSON configuration files are generated in `/path/to/fbc/config/`. A sample `alpha` realm configuration might look like the following: ```json $ more /path/to/am/config/services/global/realms/root-alpha.json { "metadata": { "realm": "/alpha", "entityType": "", "entityId": "L2FscGhh", "uid": "o=alpha,ou=services,ou=am-config", "sunServiceID": null, "objectClass": [ "sunRealmService", "top" ], "pathParams": {}, "ou": [] }, "data": { "_id": "root-alpha", "_type": { "_id": "", "name": "", "collection": false }, "active": true, "aliases": [ "alpha" ], "parentPath": "/", "name": "alpha" } } ``` ## Provide configuration files to AM 1. Stop AM or the container where it runs. 2. Copy the FBC files to the location of the *deployment layer* of your AM 8.0 configuration, by default, `/path/to/am/config/services`. Learn more about the deployment layer in [How AM reads FBC](../installation/fbc.html#how-am-reads-FBC). AM reads FBC from its home directory. You'll need access to this directory on your local host or in your Docker container. For local hosts, change to the $AM\_HOME/config/services directory: ```bash $ cd $AM_HOME/config/services ``` 3. Copy the migrated configuration into the `services` directory. ```bash $ cp /path/to/fbc/config/ $AM_HOME/config/services ``` If the directory contains default configuration, delete it before copying in your migrated configuration. ## Start AM in FBC mode When you've migrated your configuration to JSON files, you can point your upgraded AM instances to the FBC, and to the new policy and application datastores. These instructions assume a local AM server running in an Apache Tomcat container. Adjust the instructions to suit your environment. 1. Set the following Java variable to `true`: `com.sun.identity.sm.sms_object_filebased_enabled=true` 2. Pass that variable to AM in some way. For example, for Tomcat you can set the variable in the `CATALINA_OPTS` options: ```bash export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.identity.sm.sms_object_filebased_enabled=true" ``` 3. Restart AM or the container where it runs. 4. Start up and test your environment. Check that AM has the configuration you expect. You can also make changes to your configuration through the AM admin UI and check that those changes are made in the JSON configuration files. ## Troubleshoot your FBC migration Use these sections to help you troubleshoot if you encounter the following errors during a migration to FBC: > **Collapse: Failed to import /etc/am/configuration/global/DataStoreService.json** > > * Upgrade scenario > > FBC import after an Amster export > > * Error > > `user-data: [main] ERROR org.forgerock.amster.org.forgerock.openam.sdk.http.DefaultErrorHandler - Unhandled client error: [Status: 400 Bad Request] user-data: Failed to import /etc/am/configuration/global/DataStoreService.json` > > * Explanation > > The order of imported files causes the import of `DataStoreService.json` to fail because the datastore instances that it references (for external application and policy stores) have not yet been created. > > * Solution > > Import the `global/DataStoreInstance` path first, then import the remaining configuration. > **Collapse: Invalid SSO Token** > > * Upgrade scenario > > FBC import after an Amster export > > * Error > > `ERROR: Error updating properties for server null and tab sdk com.sun.identity.sm.SMSException: Invalid SSO Token: Invalid SSO Token` > > * Explanation > > Importing the CTS store properties when moving to an external CTS store fails for any subsequent import files because the CTS session is no longer valid. > > * Solution > > Import the `global/Servers` path first, then import `global/DefaultCtsDataStoreStoreProperties` and finally import the remaining configuration. > **Collapse: Data validation failed for the attribute, Secret Label** > > * Upgrade scenario > > FBC import after an Amster export of configuration prior to 7.5.0 > > * Error > > `Failed to import …​output/global/KeyStoreSecretStore/default-keystore/KeyStoreMappings/am.default.authentication.nodes.persistentcookie.encryption.json ERROR org.forgerock.openam.sdk.http.ClientErrorException: 400 Bad Request: Data validation failed for the attribute, Secret Label` > > * Explanation > > The secret mapping labels for persistent cookies changed in 7.5. As a result an attempt to import previous names fails. > > * Solution > > Before performing the import: > > 1. Remove any instances of `global/KeyStoreSecretStore/…/am.default.authentication.modules.persistentcookie.encryption.json` > > 2. Remove any instances of `global/KeyStoreSecretStore/…/am.default.authentication.modules.persistentcookie.signing.json` > > 3. Rename any instances of `global/KeyStoreSecretStore/…/am.default.authentication.nodes.persistentcookie.encryption to am.authentication.nodes.persistentcookie.encryption`. Modify the contents of each renamed file and perform the same search and replace. > > 4. When the import is complete, optionally add encryption and signing secrets in the realms in which the `PersistentCookieAuthModule` is used. The new secret mapping format is: > > `am.authentication.modules.persistentcookie.persistent-cookie-instance-name.encryption` > > `am.authentication.modules.persistentcookie.persistent-cookie-instance-name.signing` > **Collapse: services/customusernamecollector': Trying to redefine version 0.0 for path** > > * Upgrade scenario > > FBC import after an Amster export > > * Error > > `ERROR o.f.a.CrestApiProducer - ApiDescription 'frapi:openam/realm-config:2.0//services/customusernamecollector': Trying to redefine version 0.0 for path ''` > > * Explanation > > The `CustomUsernameCollectorAuth` module isn't recognized as an authentication module after the Amster import causing a failure to access the global service configuration. > > * Solution > > Before importing the upgraded Amster configuration, make sure the `'com.example.custom.CustomUsernameCollectorAuth'` is included in the list of authenticators in the `global/Authentication.json file`. Edit that file and add it as follows: > > ```json > "authenticators" : [ "com.sun.identity.authentication.modules.ad.AD", > ... > "org.forgerock.openam.authentication.modules.amster.Amster", > "org.forgerock.openam.authentication.modules.CustomUsernameCollectorAuth" ] > ``` > > | | | > | - | ----------------------------------------------------------------------------------------------------------- | > | | This example includes line breaks for legibility. In your file, the modules should all be on a single line. | > **Collapse: UMA settings aren't applied** > > * Upgrade scenario > > FBC import after an Amster export > > * Error > > `Could not create token in token data store: Operation failed: Result Code: Insufficient Access Rights` > > * Explanation > > Amster doesn't export the default server settings for UMA. It only exports specific server settings, which inherit from the default server settings. > > * Solution > > Before you import the `global/Servers/01/UmaDataStoreProperties.json` file: > > * If your deployment includes UMA configuration that is inherited from the server defaults (for example, the configuration for UMA to use external datastores), modify the inherited property to be `false` so that the specific server settings override the server defaults. > > * Alternatively, remove the specific server file so that it's not imported and doesn't override the server defaults. > **Collapse: Amster import fails for file** > > * Upgrade scenario > > FBC import after an Amster export > > * Error > > \`ERROR o.f.o.c.s.SmsJsonConverter - Invalid attribute named snmpPort specified. > > * WARN o.f.o.c.r.s.SmsSingletonProvider - ::SmsSingletonProvider > > InvalidAttributeException on Update org.forgerock.openam.core.sms.InvalidAttributeException: Invalid attribute specified.\` > > * Explanation > > Performing an Amster import from an upgraded previous version export fails for `global/Monitoring.json` because the `snmpPort` and `snmpEnabled` attributes aren't recognized. > > * Solution > > Before importing the `Monitoring.json` file, remove the `snmpPort` and `snmpEnabled` attributes and values. > **Collapse: Errors related to the keystore** > > * Upgrade scenario > > Server startup with FBC > > * Errors > > `ERROR: Unable to read private key password file` `ERROR: mapPk2Cert.JKSKeyProvider: java.io.FileNotFoundException` `ERROR: EventService.restartPSearches() Unable to start listener class com.sun.identity.sm.ldap.LDAPEventManager for data store DataStoreId…​ [CONTINUED]Unable to retrieve a connection` > > * Explanation > > These errors can be thrown when AM is unable to find the required keystores. > > * Solution > > Make sure your secret stores are configured correctly after the migration.