--- title: Trace incoming and outgoing requests description: When a user interacts with Ping Advanced Identity Software, the request can travel through multiple services before it completes. Distributed tracing lets you monitor the request flow through Ping Advanced Identity Software. component: pingam version: 8.1 page_id: pingam:monitoring:trace-requests canonical_url: https://docs.pingidentity.com/pingam/8.1/monitoring/trace-requests.html keywords: ["Distributed Tracing"] page_aliases: ["maintenance-guide:distributed-tracing.adoc", "monitoring-guide:trace-requests.adoc"] section_ids: which_requests_are_traced: Which requests are traced? understand_a_trace_object: Understand a trace object enable_and_disable_distributed_tracing: Enable and disable distributed tracing enable-tracing: Enable distributed tracing disable-tracing: Disable distributed tracing configure-distributed-tracing: Configure distributed tracing secure-trace-exports: Secure trace exports visualize_traces_with_jaeger: Visualize traces with Jaeger example_visualization_with_jaeger: Example visualization with Jaeger correlate_traces_with_audits: Correlate traces with audits --- # Trace incoming and outgoing requests When a user interacts with Ping Advanced Identity Software, the request can travel through multiple services before it completes. *Distributed tracing* lets you monitor the request flow through Ping Advanced Identity Software. Tracing provides a single view of a request's journey and makes it easier to locate bottlenecks and errors. If issues arise, tracing makes it easier to identify the service causing the problem. It's more efficient and effective than sifting through isolated logs. By identifying slow services, tracing helps you optimize application performance and reduce debugging time. This improves the end user experience as users are less likely to encounter errors or slow loading times. AM supports the [OpenTelemetry framework](https://opentelemetry.io/docs/what-is-opentelemetry/) (OTEL) for collecting distributed tracing data. ## Which requests are traced? AM supports distributed tracing for the following request types: * Incoming HTTP requests * Outgoing HTTP requests to PingIDM (Ping Advanced Identity Software deployments only) * Outgoing LDAP requests These requests are searchable and identifiable for the following LDAP operations: * ADD * MODIFY * SEARCH * DELETE * BIND * Outgoing scripting HTTP requests ## Understand a trace object This section provides a brief overview of an OTEL trace object. Learn more about trace objects in [Traces](https://opentelemetry.io/docs/concepts/signals/traces/). A *trace* represents the path of a request through an application. A trace is made up of one or more *spans*. Each span includes the following elements: * `traceId` representing the trace that the span is part of * `spanId` a unique ID for that span * `parentSpanId` the ID of the originating request The *root span* indicates the start and end of an entire operation. The `parentSpanId` of the root span is `null` because the root span isn't part of an existing trace. Subsequent spans in the trace have their own unique `spanId`. Their `traceId` is the same as that of the root span. Their `parentId` matches the `spanId` of the root span. > **Collapse: Example trace object** > > ```json > { > "spans":[ > { > "traceId":"8c3ebde938a6cf04f5bb34dd03135d45", > "spanId":"335776e2b57ee9cf", > "parentSpanId":"4065c5580cf47c7", > "name":"HTTP POST /am/json/alpha/authenticate", > "kind":1, > "startTimeUnixNano":"1718811677526761641", > "endTimeUnixNano":"1718811677526761641", > "attributes":[], > "events":[], > "Links":[], > "status":{ > "code":1 > }, > "flags":257 > } > ] > } > ``` Learn more in [Traces](https://opentelemetry.io/docs/concepts/signals/traces/) in the OpenTelemetry documentation. ## Enable and disable distributed tracing Distributed tracing is disabled by default. * [Enable distributed tracing](#enable-tracing) * [Disable distributed tracing](#disable-tracing) ### Enable distributed tracing 1. Stop AM or the container in which it runs. 2. In the `/path/to/am/config` directory, create a `deployment/trace` directory. For example: * Linux * Windows ```bash $ mkdir -p /path/to/am/config/deployment/trace ``` ```powershell PS C:\Users\Administrator> New-Item -ItemType Directory 'C:\path\to\am\config\deployment\trace' ``` 3. In the `deployment/trace` directory, create a file named `config.json` with, at least, the following contents: ```json { "tracing": { "enabled": true } } ``` Find information on additional configuration properties in [Configure distributed tracing](#configure-distributed-tracing). | | | | - | ------------------------------------------------------------------------------------------------------------------------- | | | If the content of `config.json` is invalid JSON, distributed tracing remains disabled, even if you set `"enabled": true`. | 4. Restart AM or the container in which it runs. ### Disable distributed tracing 1. Stop AM or the container in which it runs. 2. In the `/path/to/am/config/deployment/trace/config.json` file, set `"enabled": false`. 3. Restart AM or the container in which it runs. ## Configure distributed tracing The Ping Advanced Identity Software supports a common set of configuration properties for OpenTelemetry support. | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | * The stability of this configuration interface is classified as [evolving](https://docs.pingidentity.com/pingam/release-notes/stability.html#interface-stability). * Any changes to the configuration require a server restart. | To change the default OpenTelemetry configuration, add the configuration properties to your `/path/to/am/config/deployment/trace/config.json` file, for example: ```json { "tracing": { "enabled": true, "resourceAttributes": { "service.instance.id": "am-server-1" }, "exporter": { "config": { "headers": { "X-CUSTOM-HEADER": "custom-value" } } }, "spanLimits": { "maxNumberOfAttributesPerEvent": 128 } } } } ``` > **Collapse: Distributed tracing configuration properties** > > * enabled: boolean, optional > > Set to `true` to enable OpenTelemetry tracing. > > Default: `false` > > * resourceAttributes: object, optional > > A map of additional resource attributes for processing traces. Find more information in the OpenTelemetry documentation on [Semantic Attributes with SDK-provided Default Value](https://opentelemetry.io/docs/specs/semconv/resource/#semantic-attributes-with-sdk-provided-default-value). > > For example, if there are multiple AM instances in a deployment, you could set the `"service.instance.id"` resource attribute differently for each one to distinguish between them: > > ```json > { > "resourceAttributes": { > "service.instance.id": "am-server-1" > } > } > ``` > > * exporter: object, optional > > Configuration for the exporter, which pushes traces to the OpenTelemetry service: > > * type: string, optional > > Set to `otlp` for OpenTelemetry Protocol (OTLP) support. This is currently the only supported protocol. > > Default: `otlp` > > * config: object, optional > > Endpoint and timeout configuration: > > * `compressionMethod`: *enumeration, optional* > > Method used to compress trace data; either `gzip` or `none`. > > Default: `gzip` > > * `connectionTimeout`: *duration, optional* > > Time out a connection to the endpoint after this duration. > > Default: 10 seconds. > > * `endpoint`: *string, optional* > > The endpoint to publish traces to. > > For HTTPS, AM trusts the default JVM CAs. To override this, set the `-Djavax.net.ssl.trustStore` and associated JVM settings when starting AM. Learn more about the optional settings in the [Java Secure Socket Extension (JSSE) Reference Guide](https://docs.oracle.com/en/java/javase/25/security/java-secure-socket-extension-jsse-reference-guide.html). > > AM doesn't support TLS configuration for the tracing endpoint at this time. > > Default: `http://localhost:4318/v1/traces` > > * `headers`: *object, optional* > > Map of additional headers to include in the export span request. > > The following example sets the authorization header, `Authorization: Bearer ${bearer.token}`: > > ```none > "headers": { "Authorization": "Bearer ${bearer.token}" } > ``` > > * `retries`: *object, optional* > > Defines a retry policy for the export span requests. > > Default: Enabled > > * `backoffMultiplier`: *number, optional* Multiplier for the backoff wait time before retries. > > Default: 1.5 > > * `enabled`: *boolean, optional* > > Retry failed requests. > > Default: `true` > > * `initialBackoff`: *duration, optional* > > How long to wait before the first retry. > > Default: 1 second > > * `maxAttempts`: *number, optional* > > Maximum number of retries. > > Default: 5 > > * `maxBackoff`: *duration, optional* > > Maximum wait time between retries. > > Default: 5 seconds > > * `"timeout"`: *duration, optional* > > Time out a request to publish data to the endpoint after this duration. > > Default: 10 seconds. > > * `batch`: *object, optional* > > Enable and configure batch processing for trace data. > > * `compressionMethod`: *enumeration, optional* > > Method used to compress trace data; either `gzip` or `none`. > > Default: `gzip` > > * `enabled`: *boolean, optional* > > Leave batch processing enabled in deployment. > > Default: `true` > > * `exporterTimeout`: *duration, optional* > > Time out a data exporter after this duration. > > Default: 30 seconds > > * `exportUnsampledSpans`: *boolean, optional* > > Whether to report on unsampled spans. > > Default: `false` > > * `maxExportBatchSize`: *number, optional* > > Maximum number of spans in a batch. > > Default: 512 > > * `maxQueueSize`: *number, optional* > > Maximum number of spans to queue before dropping them. > > Default: 2048 > > * `scheduleDelay`: *duration, optional* > > Maximum interval between sending batches of trace data. > > Default: 50 seconds > > * `sampler`: *object, optional* > > Configuration for sampling spans. > > * `ratio`: *number, optional* > > For ratio-based types, a percentage of spans to process. > > Default: 50 (percent) > > * `type`: *string, optional* > > The sampler strategy to use is one of the following: > > * `alwaysOn`: Send every span for processing. > > * `alwaysOff`: Never send any span for processing. > > * `traceIdRatio`: Sample the specified ratio of spans, deterministically based on the trace IDs of the spans. > > * `parentBasedAlwaysOn`: Always send the span for processing if the parent span was sampled. (Default) > > * `parentBasedAlwaysOff`: Never send the span for processing if the parent span was sampled. > > * `parentBasedTraceIdRatio`: Send the specified ratio of spans for processing if the parent span was sampled. > > * `spanLimits`: *object, optional* > > Configuration for limits enforced when recording spans. > > * `maxNumberOfAttributes`: *number, optional* > > The maximum number of attributes per span. > > Default: 128 > > * `maxNumberOfAttributesPerEvent`: *number, optional* > > The maximum number of metadata items (attributes) attached to a span per event. An event is an annotation to span at a particular, meaningful point in time during the span's duration. > > Default: 128 > > * `maxNumberOfAttributesPerLink`: *number, optional* > > The maximum number of attributes per link. > > Default: 128 > > * `maxNumberOfEvents`: *number, optional* > > The maximum number of events per span. > > Default: 128 > > * `maxNumberOfLinks`: *number, optional* > > The maximum number of links per span. Links associate the current span with one or more other spans. > > Default: 128 ## Secure trace exports To protect trace exports, configure a secure connection between AM and the trace collector. These steps assume you're using the OpenTelemetry (OTel) Collector, but you can choose any trace collector. Adjust the steps accordingly. Configuring a secure connection involves the following steps: 1. Configure the collector to receive traces over HTTPS. Specify the path to your collector's TLS certificate and private key and, optionally, the path to the CA certificate. If you're using the OTel collector, read OTel's collector documentation for information on [configuring TLS](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md). 2. In the `/path/to/am/config/deployment/trace/config.json` file, make sure the exporter endpoint references HTTPS. For example: ```json { "tracing": { "enabled": true, "exporter": { "config": { "endpoint": "https://otelcol.localtest.me:4318/v1/traces" } } } } ``` 3. Add the trace collector's TLS certificate to the AM truststore. For example: ```bash $ keytool \ -importcert \ -file /path/to/cert_file.pem \ -keystore /path/to/am/security/keystores/truststore ``` 4. Restart AM or the container in which it runs. ## Visualize traces with Jaeger You can use [Jaeger](https://www.jaegertracing.io/docs/1.56/getting-started/) to collect trace data directly from AM (or from your chosen telemetry collector) and to visualize that trace data. ### Example visualization with Jaeger This example assumes a local AM deployment in Apache Tomcat. 1. Start Jaeger. Jaeger runs in a Docker container. Start Jaeger with a command similar to the following: ```bash docker run --rm --name jaeger \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 9411:9411 \ jaegertracing/jaeger:2.0.0 \ --set receivers.otlp.protocols.http.endpoint=0.0.0.0:4318 ``` 2. Go through an AM authentication flow. This example authenticates the `amAdmin` user: ```bash * curl \ --request POST \ --header "Content-Type: application/json" \ --header "X-OpenAM-Username: amadmin" \ --header 'X-OpenAM-Password: password' \ --header "Accept-API-Version: resource=2.0, protocol=1.0" \ "https://am.example.com:8443/am/json/realms/root/authenticate"* ``` 3. Navigate to Jaeger's UI at . 4. Under Service select `am` (or the context path where you deployed AM) then click Find Traces. ![jaeger ui](../maintenance/_images/jaeger-ui.png) 5. Click the trace for the `HTTP POST` request to the `authenticate` endpoint to view the traces for the authentication flow: ![jaeger post request](../maintenance/_images/jaeger-post-request.png) Note the `forgerock.transaction_id 8156ce8c-c5f4-4f59-ba59-9dd41d654f68-54971`. This is the ID you'll use to correlate the trace with the AM audit logs. ## Correlate traces with audits Distributed tracing doesn't change the AM audit logs in any way. However, when you enable distributed tracing, you can enrich the traces to include audit transaction IDs in their metadata. If a request includes a transaction identifier and a *span* is created for that request, the span metadata is enriched with the transaction identifier. This lets you correlate requests between traces and audit logs to determine which requests are taking longer than anticipated and identify any bottlenecks. In the previous example, the `forgerock.transaction_id` was `8156ce8c-c5f4-4f59-ba59-9dd41d654f68-54971` for the authentication request. The entry in the `authentication.audit.json` file for that request is as follows: ```json { "_id": "8156ce8c-c5f4-4f59-ba59-9dd41d654f68-54986", "timestamp": "2024-11-21T12:30:22.561Z", "eventName": "AM-LOGIN-COMPLETED", "transactionId": "8156ce8c-c5f4-4f59-ba59-9dd41d654f68-54971", "trackingIds": [ "8156ce8c-c5f4-4f59-ba59-9dd41d654f68-54974" ], "userId": "id=amadmin,ou=user,dc=am,dc=example,dc=com", "principal": [ "amadmin" ], ... } ``` Note the correlation between the `forgerock.transaction_id` in the trace and the `transactionId` in the log entry.