--- title: Distributed tracing description: Distributed tracing helps you monitor and observe system requests as they flow through IDM. A request is tracked and analyzed with a unique identifier (ID) used to troubleshoot requests that show errors or performance issues. This unique ID remains with a transaction as it interacts with microservices, containers, and infrastructure. component: pingidm version: 8.1 page_id: pingidm:monitoring-guide:distributed-tracing canonical_url: https://docs.pingidentity.com/pingidm/8.1/monitoring-guide/distributed-tracing.html keywords: ["Monitoring", "Distributed tracing"] section_ids: tracing-benefits: Tracing benefits idm_and_opentelemetry: IDM and OpenTelemetry request_types_supported_in_idm: Request types supported in IDM understand-trace-object: Understand a trace object enable-tracing: Enable distributed tracing disable-tracing: Disable distributed tracing configure-distributed-tracing: Configure distributed tracing viz-otel-traces: Visualize traces with a trace collector ex-viz-trace-collector: Example visualization with a trace collector --- # Distributed tracing | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Distributed tracing is an [Evolving](../release-notes/appendix-interface-stability.html) feature in PingIDM. It's subject to change without notice, even in a minor or maintenance release. | Distributed tracing helps you monitor and observe system requests as they flow through IDM. A request is tracked and analyzed with a unique identifier (ID) used to troubleshoot requests that show errors or performance issues. This unique ID remains with a transaction as it interacts with microservices, containers, and infrastructure. When a user interacts with the Ping Identity Platform, the request can travel through multiple services before it completes. Distributed tracing lets you monitor the request flow through the Ping Identity Platform. ## Tracing benefits Distributed tracing makes it easier to: * Provide a single view of a request's journey * Locate bottlenecks and errors * Identify slow services * Optimize application performance and reduce debugging time * Improve the end-user experience ## IDM and OpenTelemetry IDM supports the [OpenTelemetry framework](https://opentelemetry.io/docs/what-is-opentelemetry/) (OTEL) for collecting [distributed tracing](#distributed-tracing) and [logging](opentelemetry-logging.html) data. OpenTelemetry handles: * Generation * Collection * Management * Export of telemetry | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | OpenTelemetry doesn't handle telemetry storage and visualization. You can use a trace collector, such as [Jaeger](https://www.jaegertracing.io/), to collect and visualize trace data. | ## Request types supported in IDM IDM supports distributed tracing for the following request types: * Incoming HTTP requests * Outgoing HTTP requests to PingAM (Ping Identity Platform deployments only) * Outgoing LDAP requests These requests are searchable and identifiable for the following LDAP operations: * ADD * MODIFY * SEARCH * DELETE * BIND * Outgoing scripting HTTP requests | | | | - | ----------------------------------------------------------------------------------------------------------------------------------- | | | Outgoing scripting HTTP requests must use the `openidm.action` function to make an external call using IDM's external REST service. | ## Understand a trace object A *trace* represents the path of a request through an application. A trace is made up of one or more *spans*. Each span includes the following elements: * `traceId`: Representing the trace that the span is a part of * `spanId`: A unique ID for that span * `parentSpanId`: The ID of the originating request The *root span* indicates the start and end of an entire operation. The `parentSpanId` of the root span is `null` because the root span isn't part of an existing trace. Subsequent spans in the trace have their own unique `spanId`. The `traceId` is the same as that of the root span. The `parentId` matches the `spanId` of the root span. Learn more in [Traces](https://opentelemetry.io/docs/concepts/signals/traces/) in the OpenTelemetry documentation. ## Enable distributed tracing | | | | - | ------------------------------------------- | | | Distributed tracing is disabled by default. | To enable distributed tracing: 1. In the `/path/to/openidm` directory, create a `/trace` directory, for example, where you can place the tracing configuration file. | | | | - | ------------------------------------------------------------------------------------------------------------------------------ | | | The configuration file isn't required to be in the `/conf` directory. You can place this file in any location readable by IDM. | 2. In the `/trace` directory, for example, create an OTEL configuration JSON file with the following information and set `"enabled": true`: ```json { "tracing": { "enabled": true, "exporter": { "type": "otlp", "config": { "endpoint": "http://localhost:4318/v1/traces" } } } } ``` You can find information on additional configuration properties in [Configure distributed tracing](#configure-distributed-tracing). | | | | - | ---------------------------------------------------------------------------------------------------------------------------------- | | | If the content of the configuration file is invalid JSON, distributed tracing remains disabled, even if you set `"enabled": true`. | 3. IDM uses the environment variable `OPENIDM_TRACING_CONFIG_PATH`. Set this environment variable to point to the configuration file in `/path/to/openidm` or to any location readable by IDM, for example: ``` OPENIDM_TRACING_CONFIG_PATH=/path/to/openidm/trace/tracing.json ``` 4. After you create the OTEL configuration file in the directory that you choose and point the environment variable to the OTEL configuration file, [start IDM](../install-guide/chap-install.html#run-openidm). Starting IDM launches a service that reads the OTEL configuration file and monitors it for changes to perform runtime updates to the distributed tracing service. ### Disable distributed tracing To disable distributed tracing: In the OTEL configuration file, set `"enabled": false`. After you've changed the configuration file, the service that detects configuration changes relaunches the distributed tracing service in a disabled state. ## Configure distributed tracing The Ping Identity Platform supports a common set of configuration properties for OpenTelemetry support. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | * The stability of this configuration interface is classified as [Evolving](../release-notes/appendix-interface-stability.html). * Any changes to the configuration require a server restart. | To change the default OpenTelemetry configuration, add the configuration properties to your configuration file, for example: ```json { "tracing": { "enabled": true, "resourceAttributes": { "service.instance.id": "idm-server-1" }, "exporter": { "config": { "headers": { "X-CUSTOM-HEADER": "custom-value" } } }, "spanLimits": { "maxNumberOfAttributesPerEvent": 128 } } } ``` > **Collapse: Distributed tracing configuration properties** > > * enabled: boolean, optional > > Set to `true` to enable OpenTelemetry tracing. > > Default: `false` > > * resourceAttributes: object, optional > > A map of additional resource attributes for processing traces. Find more information in the OpenTelemetry documentation on [Semantic Attributes with SDK-provided Default Value](https://opentelemetry.io/docs/specs/semconv/resource/#semantic-attributes-with-sdk-provided-default-value). > > For example, if there are multiple Ping Identity Platform instances in a deployment, you could set the `"service.instance.id"` resource attribute differently for each one to distinguish between them: > > ```json > { > "resourceAttributes": { > "service.instance.id": "idm-server-1" > } > } > ``` > > * exporter: object, optional > > Configuration for the exporter, which pushes traces to the OpenTelemetry service: > > * type: string, optional > > Set to `otlp` for OpenTelemetry Protocol (OTLP) support. This is currently the only supported protocol. > > Default: `otlp` > > * config: object, optional > > Endpoint and timeout configuration: > > * `compressionMethod`: *enumeration, optional* > > Method used to compress trace data; either `gzip` or `none`. > > Default: `gzip` > > * `connectionTimeout`: *duration, optional* > > Time out a connection to the endpoint after this duration. > > Default: 10 seconds. > > * `endpoint`: *string, optional* > > The endpoint to publish traces to. > > For HTTPS, IDM trusts the default JVM CAs. To override this, set the `-Djavax.net.ssl.trustStore` and associated JVM settings when starting IDM. Learn more about the optional settings in the [Java Secure Socket Extension (JSSE) Reference Guide](https://docs.oracle.com/en/java/javase/21/security/java-secure-socket-extension-jsse-reference-guide.html). > > | | | > | - | ---------------------------------------------------------------------------- | > | | IDM doesn't support TLS configuration for the tracing endpoint at this time. | > > Default: `http://localhost:4318/v1/traces` > > * `headers`: *object, optional* > > Map of additional headers to include in the export span request. > > The following example sets the authorization header, `Authorization: Bearer ${bearer.token}`: > > ```none > "headers": { "Authorization": "Bearer ${bearer.token}" } > ``` > > * `retries`: *object, optional* > > Defines a retry policy for the export span requests. > > Default: Enabled > > * `backoffMultiplier`: *number, optional* Multiplier for the backoff wait time before retries. > > Default: 1.5 > > * `enabled`: *boolean, optional* > > Retry failed requests. > > Default: `true` > > * `initialBackoff`: *duration, optional* > > How long to wait before the first retry. > > Default: 1 second > > * `maxAttempts`: *number, optional* > > Maximum number of retries. > > Default: 5 > > * `maxBackoff`: *duration, optional* > > Maximum wait time between retries. > > Default: 5 seconds > > * `timeout`: *duration, optional* > > Time out a request to publish data to the endpoint after this duration. > > Default: 10 seconds. > > * `batch`: *object, optional* > > Enable and configure batch processing for trace data. > > * `compressionMethod`: *enumeration, optional* > > Method used to compress trace data; either `gzip` or `none`. > > Default: `gzip` > > * `enabled`: *boolean, optional* > > Leave batch processing enabled in deployment. > > Default: `true` > > * `exporterTimeout`: *duration, optional* > > Time out a data exporter after this duration. > > Default: 30 seconds > > * `exportUnsampledSpans`: *boolean, optional* > > Whether to report on unsampled spans. > > Default: `false` > > * `maxExportBatchSize`: *number, optional* > > Maximum number of spans in a batch. > > Default: 512 > > * `maxQueueSize`: *number, optional* > > Maximum number of spans to queue before dropping them. > > Default: 2048 > > * `scheduleDelay`: *duration, optional* > > Maximum interval between sending batches of trace data. > > Default: 50 seconds > > * `sampler`: *object, optional* > > Configuration for sampling spans. > > * `ratio`: *number, optional* > > For ratio-based types, a percentage of spans to process. > > Default: 50 (percent) > > * `type`: *string, optional* > > The sampler strategy to use is one of the following: > > * `alwaysOn`: Send every span for processing. > > * `alwaysOff`: Never send any span for processing. > > * `traceIdRatio`: Sample the specified ratio of spans deterministically based on the trace IDs of the spans. > > * `parentBasedAlwaysOn`: Always send the span for processing if the parent span was sampled. (Default) > > * `parentBasedAlwaysOff`: Never send the span for processing if the parent span was sampled. > > * `parentBasedTraceIdRatio`: Send the specified ratio of spans for processing if the parent span was sampled. > > * `spanLimits`: *object, optional* > > Configuration for limits enforced when recording spans. > > * `maxNumberOfAttributes`: *number, optional* > > The maximum number of attributes per span. > > Default: 128 > > * `maxNumberOfAttributesPerEvent`: *number, optional* > > The maximum number of metadata items (attributes) attached to a span per event. An event is an annotation to span at a particular, meaningful point in time during the span's duration. > > Default: 128 > > * `maxNumberOfAttributesPerLink`: *number, optional* > > The maximum number of attributes per link. > > Default: 128 > > * `maxNumberOfEvents`: *number, optional* > > The maximum number of events per span. > > Default: 128 > > * `maxNumberOfLinks`: *number, optional* > > The maximum number of links per span. Links associate the current span with one or more other spans. > > Default: 128 ## Visualize traces with a trace collector Trace collectors work alongside IDM, allowing the service to offload data quickly. A collector manages retries, batching, encryption, and data filtering. You can use a trace collector to collect trace data from the OpenTelemetry Collector and visualize that data. ### Example visualization with a trace collector This example assumes a local IDM deployment. 1. Start the trace collector. 2. Send a request against IDM: ```bash curl --location 'http://localhost:8080/openidm/schema/managed/user' \ --header 'x-openidm-username: openidm-admin' \ --header 'x-openidm-password: openidm-admin' ``` 3. Query the trace collector's localhost URL for the request: ```bash curl --location 'http://localhost:portnumber/api/traces?service=idm' ``` 4. You should receive a query response similar to the following: ```json { "data": [ { "traceID": "09cb4130f4c8803011b3996f8bda6b8c", "spans": [ { "traceID": "09cb4130f4c8803011b3996f8bda6b8c", "spanID": "e49cd1d70d65502d", "operationName": "GET /openidm", "references": [], "startTime": 1741892475358333, "duration": 165774, "tags": [ { "key": "forgerock.transaction_id", "type": "string", "value": "c246e9ea-e596-4a99-a0eb-5072806cefe3-890" }, { "key": "http.request.method", "type": "string", "value": "GET" }, { "key": "network.protocol.name", "type": "string", "value": "http" }, { "key": "otel.scope.name", "type": "string", "value": "idm" }, { "key": "span.kind", "type": "string", "value": "server" }, { "key": "url.full", "type": "string", "value": "http://localhost:8080/openidm/schema/managed/user" }, { "key": "url.path", "type": "string", "value": "/openidm/schema/managed/user" } ], "logs": [], "processID": "p1", "warnings": null } ], ... } ] } ```