--- title: Read audit logs using REST description: "Estimated time to complete: 30 minutes" component: pingoneaic page_id: pingoneaic:use-cases:use-case-audit-logging canonical_url: https://docs.pingidentity.com/pingoneaic/use-cases/use-case-audit-logging.html keywords: ["Implementation Guide", "Use Case", "REST API", "Debug", "Audit", "Logs"] page_aliases: ["implementation:use-case-audit-logging.adoc"] section_ids: audit-logging-description: Description audit-logging-goals: Goals audit-logging-prereqs: Prerequisites audit-logging-tasks: Tasks audit-logging-task-1: "Task 1: Create an API key and secret, and retrieve log sources" audit-logging-task-2: "Task 2: Get logs" audit-logging-task-3: "Task 3: Filter log results" audit-logging-validation: Validation audit-logging-steps: Steps audit-logging-explore-further: Explore further audit-logging-reference-material: Reference material --- # Read audit logs using REST ## Description Estimated time to complete: 30 minutes *(tooltip: This assumes you complete the prerequisites beforehand.)* In this use case, you examine audit logs to investigate user behavior, and you use debug logs to investigate system behavior. ### Goals After completing this use case, you will know how to do the following: * Retrieve logs * Filter log results * Tail logs * View logs for a specific request ## Prerequisites Before you start work on this use case, ensure that you have these prerequisites: * A basic understanding of Advanced Identity Cloud logging sources * Access to your Advanced Identity Cloud development environment as a tenant administrator * General understanding of how to run commands from the terminal window * The `curl` command installed on your system ## Tasks ### Task 1: Create an API key and secret, and retrieve log sources In this task, you create an API key and an API secret in the Advanced Identity Cloud admin console; you'll need these values to call the `/logs` endpoint. Then, you call the `/logs/sources` endpoint to retrieve log sources. 1. Get an API key and an API secret: 1. Log in to the Advanced Identity Cloud admin console as an administrator. 2. In the Advanced Identity Cloud admin console, access the Tenant menu (upper right). 3. Select Tenant Settings. 4. Click Global Settings. 5. On the Global Settings tab, click Log API Keys. 6. Click + New Log API Key 7. Provide a name for the key. 8. Click Create key. The UI displays a dialog box containing the new keys: ![Key successfully created message. You have options to copy the api\_key\_id and api\_key\_secret.](../developer-docs/_images/log-api-key.png) 9. Copy the api\_key\_id and api\_key\_secret values from the dialog box and save them using a text editor. You'll need to access these two values whenever you make an API call in this use case. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Store the values in a safe place, and do not share them with anyone else. The API key and API secret values provide access to administrative endpoints in Advanced Identity Cloud. | 10. After you've saved the api\_key\_id and api\_key\_secret values, click Done. You won't be able to view the api\_key\_secret value again after you click Done. If you didn't save this value, you'll need to repeat the preceding steps to create another API key and secret. 2. Get your tenant FQDN. In the Advanced Identity Cloud admin console, go to [icon: cog, set=fa]Tenant Settings > Details. Your tenant FQDN is labeled Tenant Name. 3. Advanced Identity Cloud makes browsing the logs easier by storing them in various sources. Learn more about logging sources in [Log sources](../tenants/audit-debug-log-sources.html). List the logging sources available in Advanced Identity Cloud: 1. Open a terminal window. 2. Run a `curl` command\[[1](#_footnotedef_1 "View footnote.")] to list the log sources. Specify variables in the `curl` command as follows: * \ — The api\_key\_id value from step 1 * \ — The api\_key\_secret value from step 1 * \ — Your Advanced Identity Cloud tenant FQDN from step 2 ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ 'https:///monitoring/logs/sources' ``` The terminal displays output similar to this: ```none {"result":["am-access","am-activity","am-authentication","am-config","am-core","am-everything","idm-access","idm-activity","idm-authentication","idm-config","idm-core","idm-everything","idm-recon","idm-sync"],"resultCount":14,"pagedResultsCookie":null,"totalPagedResultsPolicy":"NONE","totalPagedResults":1,"remainingPagedResults":0} ``` 4. Rerun the `curl` command to list the log sources, but this time, filter the command's JSON output through the `json_pp` command\[[2](#_footnotedef_2 "View footnote.")] to make the output easier to read: ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ 'https:///monitoring/logs/sources' | json_pp { "pagedResultsCookie" : null, "remainingPagedResults" : 0, "result" : [ "am-access", "am-activity", "am-authentication", "am-config", "am-core", "am-everything", "idm-access", "idm-activity", "idm-authentication", "idm-config", "idm-core", "idm-everything", "idm-recon", "idm-sync" ], "resultCount" : 14, "totalPagedResults" : 1, "totalPagedResultsPolicy" : "NONE" } ``` ### Task 2: Get logs In this task, you retrieve all available log entries from a log source by calling the `logs` endpoint. Then you retrieve the tail end of a log source by calling the `logs/tail` endpoint. 1. Run a `curl` command to retrieve the `am-authentication` log source. Specify variables in the `curl` command as follows: * \ — The api\_key\_id value you got in Task 1 * \ — The api\_key\_secret value you got in Task 1 * \ — Your Advanced Identity Cloud tenant FQDN ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-authentication' \ 'https:///monitoring/logs' ``` The `curl` command displays all available log entries in the `am-authentication` log source. A query like this one, which usually displays a large amount of log data, is not particularly useful as it is difficult to work with the output without a log analysis tool. 2. Retrieve the `am-authentication` log source using the `logs/tail` endpoint. Run a similar `curl` call, but instead of calling the `logs` endpoint, call the `logs/tail` endpoint and filter the output through the `json_pp` command: ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-authentication' \ 'https:///monitoring/logs/tail' | json_pp ``` A smaller amount of output displays. Your call may not return any log entries at all if there hasn't been any recent authentication activity. When you call the `/logs/tail` endpoint without specifying any parameters other than the log source, the call returns log entries from the last 15 seconds of Advanced Identity Cloud activity. If the call to the `/logs/tail` endpoint didn't return any log entries, log in to the Advanced Identity Cloud admin console, wait a few seconds for Advanced Identity Cloud to write log entries, and call the `/logs/tail` endpoint again; several log entries should display. 3. Call the `logs/tail` endpoint repeatedly to see changes to a log source since the last time you called the endpoint. When you specify the `_pagedResultsCookie` parameter when calling the `logs/tail` endpoint, it returns only the changes that have been made to the log source since the last time you called the endpoint. In this step, you'll use the `am-everything` log source; more entries are written to this log source than to the `am-authentication` log source: 1. Retrieve the `am-everything` log source using the `logs/tail` endpoint: ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-everything' \ 'https:///monitoring/logs/tail' | json_pp ``` 2. Review the start of the output from the call to the `logs/tail` endpoint. You'll see a value for the `pagedResultsCookie` key: ```none "pagedResultsCookie" : "eyJfc29ydEtleXM...", "remainingPagedResults" : -1, "result" : [], "resultCount" : 0, "totalPagedResults" : -1, "totalPagedResultsPolicy" : "NONE" } ``` 3. Save the `pagedResultsCookie` key's value using a text editor. You'll need this value for subsequent calls to the `/logs/tail` endpoint. 4. Retrieve the `am-everything` log source again using the `logs/tail` endpoint. This time, specify the `_pagedResultsCookie` parameter in your `curl` call so that only the log entries written to the log source since your previous call to the `/logs/tail` endpoint are displayed: ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-everything' \ --data '_pagedResultsCookie=' \ 'https:///monitoring/logs/tail' | json_pp ``` 5. Run the same `curl` command several times to see the log entries written to the `am-everything` log source since the call from which you saved the `pagedResultsCookie` value. Additional log entries should be displayed for each call, but the output depends on the level of activity in your Advanced Identity Cloud tenant. ### Task 3: Filter log results In this task, you'll filter log data using the `_queryFilter` parameter so that you get a limited amount of log entries from queries. You'll run a variety of queries to gain experience with filtering. 1. To formulate query filters, you first need to understand the structure of a log query result so that you can construct a query filter expression: 1. Run the same query to retrieve the `am-authentication` log source you ran in Task 2: ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-authentication' \ 'https:///monitoring/logs' | json_pp ``` If you don't get any results from your query, log in to the Advanced Identity Cloud admin console, wait a few seconds for Advanced Identity Cloud to write log entries, and then rerun your `curl` call. You should get results after doing this. 2. Examine the value of one of the `payload` keys the `curl` command returned. The JSON output should be similar to *if not exactly the same as* the following; different types of log entries have different JSON objects: ```none { "payload" : { "_id" : "d6741f27-47b2-4047-8c61-09fe846b7d93-70338", "component" : "Authentication", "entries" : [ { "info" : { "authLevel" : "0", "displayName" : "MFA Required?", "nodeId" : "6ed9e2ac-bc5a-49b0-ac81-3e33c4ccfa1d", "nodeOutcome" : "Optional", "nodeType" : "ScriptedDecisionNode", "treeName" : "FRSecondFactor" } } ], "eventName" : "AM-NODE-LOGIN-COMPLETED", "level" : "INFO", "principal" : [ "pat@example.com" ], "realm" : "/", "source" : "audit", "timestamp" : "2023-09-11T18:16:33.923Z", "topic" : "authentication", "trackingIds" : [ "d6741f27-47b2-4047-8c61-09fe846b7d93-70286" ], "transactionId" : "4cbed6e6-6b10-4a06-b978-b95d25b7c254-request-2/0" }, "source" : "am-authentication", "timestamp" : "2023-09-11T18:16:33.92361169Z", "type" : "application/json" } ``` You specify JSON keys in log query filter expressions, using a syntax where objects in the JSON hierarchy are separated by forward slashes. In the example JSON output above: * The `payload` object has a nested object named `eventName`; a query filter to retrieve log entries for a specific log event would compare `/payload/eventName` to the log event name. * The `payload` object has a nested object named `principal`; a query filter to retrieve log entries for a specific user (also known as a principal) would compare `/payload/principal` to the user's name. Learn more about query filter syntax in [Filter expressions](../developer-docs/crest/query.html#crest-query-queryFilter). For examples of query filters, refer to the table in [Filter log results](../tenants/audit-debug-logs-pull.html#filter-log-results). 2. Run a `curl` command to retrieve log entries for your authentication activity. Specify variables in the `curl` command as follows: * \ — The api\_key\_id value you got in Task 1 * \ — The api\_key\_secret value you got in Task 1 * \ — The user ID you use to log into Advanced Identity Cloud * \ — Your Advanced Identity Cloud tenant FQDN ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-authentication' \ --data-urlencode '_queryFilter=/payload/principal eq ""' \ 'https:///monitoring/logs' ``` If you don't get any results from your query, log in to the Advanced Identity Cloud admin console, wait a few seconds for Advanced Identity Cloud to write log entries, and then rerun your `curl` call. You should get results after doing this. 3. Run a `curl` command to retrieve log entries that indicate you have completed an authentication journey. Specify variables in the `curl` command as follows: * \ — The api\_key\_id value you got in Task 1 * \ — The api\_key\_secret value you got in Task 1 * \ — The user ID you use to log into Advanced Identity Cloud * \ — Your Advanced Identity Cloud tenant FQDN ```none $ curl --get \ --header 'x-api-key: ' \ --header 'x-api-secret: ' \ --data 'source=am-authentication' \ --data-urlencode '_queryFilter=/payload/principal eq "" and /payload/eventName eq "AM-TREE-LOGIN-COMPLETED"' \ 'https:///monitoring/logs' ``` If you don't get any results from your query, log in to the Advanced Identity Cloud admin console, wait a few seconds for Advanced Identity Cloud to write log entries, and then rerun your `curl` call. You should get results after doing this. ## Validation Now that you've had some practice writing API calls to retrieve log entries, you're ready to validate your skills by retrieving all the log entries for a request to Advanced Identity Cloud. All log events for an external request to Advanced Identity Cloud, such as authentication, are assigned the same unique transaction ID. Reviewing multiple log entries that a single request generates can be valuable when troubleshooting system behavior. ### Steps In this validation, demonstrate the ability to get a set of log entries for an external request to Advanced Identity Cloud. First, authenticate to the Advanced Identity Cloud admin console. Then, call the log API to obtain the transaction ID. Finally, run filtered queries to retrieve all the log entries for your authentication. 1. Log in to the Advanced Identity Cloud admin console. 2. Run a `curl` command to retrieve log entries from the `am-authentication` log source. 3. Find one of the log entries for your authentication attempt in the `curl` command output. 4. Find the transaction ID in this log record. The key-value pair to locate is similar to this example: ```none "transactionId" : "4cbed6e6-6b10-4a06-b978-b95d25b7c254-request-2/0" ``` 5. Run another `curl` command to retrieve all the log entries for your transaction from the `am-authentication` log source. You should retrieve multiple log entries. Tips: * If you use the `logs/tail` endpoint, you may not get any log entries back. Remember, the `logs/tail` endpoint retrieves log entries from the only last 15 seconds of Advanced Identity Cloud activity. * When constructing your `curl` command, use `/payload/transactionId` as the JSON key for your query filter. 6. Run another `curl` command to retrieve all the log entries for your transaction from all the log sources. Tips: * Use the `am-everything` log source in your `curl` command. * Expect a large number of log entries to be returned in the command output. ## Explore further ### Reference material | **Reference** | **Learning Asset** | **Description** | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Getting started with the ForgeRock Identity Cloud REST API: Part 8 - Auditing and monitoring](https://community.forgerock.com/t/getting-started-with-the-forgerock-identity-cloud-rest-api-part-8-auditing-and-monitoring/3383) | Community | Examples of running Advanced Identity Cloud auditing and monitoring requests using both Postman and `curl` commands. | | [Stream logs to an external monitoring tool](../tenants/audit-debug-logs-push.html) | Documentation | Monitor log events in real time using your preferred SIEM or event monitoring solution, such as Splunk or an OpenTelemetry-compatible SIEM like Grafana Cloud, Datadog, or New Relic. | *** [1](#_footnoteref_1). All the example commands in this use case are written using Z shell syntax. You may need to adjust your command syntax slightly if you are using a different shell.[2](#_footnoteref_2). If you're running on Microsoft Windows, you'll need to use an alternative command such as the `jq` command for JSON formatting.