--- title: Distributed tracing description: Learn more about how PingAccess supports distributed tracing. component: pingaccess version: 9.1 page_id: pingaccess:troubleshooting:pa_distributed_tracing canonical_url: https://docs.pingidentity.com/pingaccess/9.1/troubleshooting/pa_distributed_tracing.html revdate: June 17, 2026 section_ids: what-is-distributed-tracing: What is distributed tracing? supported-request-types: Supported request types distributed-tracing-components: Distributed tracing components visualizing-traces: Visualizing traces configuring-distributed-tracing: Configuring distributed tracing configure-distributed-tracing-with-property-files: Configure distributed tracing with property files steps: Steps example: Example: configure-distributed-tracing-with-environment-variables: Configure distributed tracing with environment variables steps-2: Steps example-2: Example: --- # Distributed tracing Distributed tracing provides visibility into the full path a request takes in a distributed system, helping to instrument, collect, and export telemetry data. Telemetry data can make it easier to identify the root cause of performance issues or errors, especially when investigating backchannel calls. Ping Identity supports the [OpenTelemetry framework (OTEL)](https://opentelemetry.io/docs/what-is-opentelemetry/) for collecting distributed tracing data. PingAccess uses [OpenTelemetry Protocol (OTLP)](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.51.0/oteps/0035-opentelemetry-protocol.md) to send distributed traces to a backend service such as [Jaeger](https://www.jaegertracing.io/) for collection, storage, and visualization. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------- | | | Distributed tracing is an evolving feature in PingAccess. It's subject to change without notice, even in a minor or maintenance release. | ## What is distributed tracing? In a distributed system, requests pass through multiple services hosted on multiple servers. Distributed tracing shows you how an incoming request was processed across all servers and services, including: * Which servers and services the request went through. * How much time each service took to process its part of the request. * How the services are connected. * What the failure point was in case of a request failure. ## Supported request types PingAccess supports distributed tracing for the following request types: * Incoming HTTP requests * Outgoing HTTP requests * Internal Java database connectivity (JDBC) requests ## Distributed tracing components * Traces A trace represents the path of a request through an application. A trace is made up of one or more spans. Learn more about traces in the [OpenTelemetry documentation](https://opentelemetry.io/docs/concepts/signals/traces/). * Spans A span is a segment of a request journey. It represents a unit of work or an operation within a service. Each span includes the following elements: * `traceId` represents the trace that the span is a part of. * `spanId` is a unique ID for the span. * `parentSpanId` is the ID of the originating request. * Root span The root span indicates the start and end of an entire operation. The `parentSpanId` of the root span is null because the root span isn't part of an existing trace. Subsequent spans in the trace have their own unique `spanId`. Their `traceId` is the same as that of the root span, and their `parentSpanId` matches the `spanId` of the root span. * OpenTelemetry OpenTelemetry is an open-source observability framework for instrumenting, generating, collecting, and exporting telemetry data. It provides a standardized way to capture distributed traces across different services and platforms. It doesn't provide a backend for storing or analyzing telemetry data. Learn more in the [OpenTelemetry documentation](https://opentelemetry.io/docs/what-is-opentelemetry/). ## Visualizing traces PingAccess can push traces to an [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otel/protocol/) endpoint over HTTP. Any backend that supports OTLP and HTTP can be used to collect and visualize the traces. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Try the [Jaeger tracing All-in-one Docker image](https://www.jaegertracing.io/docs/2.11/getting-started/) to capture exported spans. By default, Jaeger stores the spans in memory, but you can configure Jaeger to send the spans to various persistent datastores external to the Docker image. | PingAccess includes an `exchangeId` tag on each operation it performs. You can use this tag to compare a specific span with a corresponding entry in the PingAccess log. Debug log level entries can provide additional insight after you use a tool like Jaeger to identify where the problem might lie. ## Configuring distributed tracing Enable distributed tracing and configure relevant OpenTelemetry settings using one of the following methods: * Edit the `run.properties` and `opentelemetry.properties` files directly. * Use environment variables to set the properties in the `run.properties` and `opentelemetry.properties` files. Properties set by environment variables take precedence over any property values specified in the `opentelemetry.properties` file. | | | | - | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | If you're [running PingAccess as a Windows service](../installing_and_uninstalling_pingaccess/pa_managing_pa_as_a_windows_service.html), you must reinstall the service for changes to the `pa.enable.distributed.tracing` property to take effect. This limitation applies regardless of the configuration method you use. | * With the property files * With environment variables ## Configure distributed tracing with property files ### Steps 1. Enable distributed tracing: 1. Open the `/conf/run.properties` file in a text editor. 2. Set the `pa.enable.distributed.tracing` property to `true`. #### Example: `pa.enable.distributed.tracing=true` 2. To configure OpenTelemetry properties, open the `/conf/opentelemetry.properties` file in a text editor. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The following table lists the default properties in `opentelemetry.properties`. You can find the full list of configurable OpenTelemetry properties in the [OpenTelemetry SDK documentation](https://opentelemetry.io/docs/languages/java/configuration/). | > **Collapse: Default OpenTelemetry properties** > > | Property | Description | > | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | > | `otel.service.name` (required) | Specify the name of your application or service. The default value is PingAccess. If running PingAccess in a clustered environment, it might be helpful to give each node a different otel.service.name so you can tell them apart if one is having latency issues. | > | `otel.exporter.otlp.endpoint` (required) | Specify the endpoint for your OpenTelemetry collector or backend:- If using OTLP or gRPC, use port 4317. > > For example, `http://localhost:4317`. > > - If using either OTLP or HTTP with protobuf, use port 4318.The default value is `http://localhost:4318`. | > | `otel.exporter.otlp.protocol` (required) | Specify the OTLP transport protocol to use for all telemetry data. Valid values include:- `http/protobuf` (default) > > PingAccess recommends using this setting for its simplicity and compatibility with firewalls. > > If using `http/protobuf`, make sure `otel.exporter.otlp.endpoint` is set to port 4318. > > - `grpc` to use OTLP or gRPC. > > If using `grpc`, make sure `otel.exporter.otlp.endpoint` is set to port 4317. > > - `http/json` to use either OTLP or HTTP with JSON. > > Review your backend OTEL consumer documentation for recommendations on which port to use for JSON. | > | `otel.resource.attributes` (optional) This property is commented out by default. | Optionally define static environment metadata for all telemetry data. Use the `key1=val,key2=val` format.For example, to add the environment, deployment zone, and host role, set `otel.resource.attributes=environment=dev,deployment.zone=us-east-1a,host.role=web-server`. | > | `otel.instrumentation.` (optional) This property is commented out by default. | You can use this property to disable specific instrumentation modules if they cause noise or conflicts. The value after otel.instrumentation must match the instrumentation name. For example, to disable JDBC instrumentation, set otel.instrumentation.jdbc.enabled=false. | > | `otel.javaagent.logging` (required) | You can use this property to control OpenTelemetry agent logging. The default value is `application`. Ping Identity recommends using the default value because it integrates with the PingAccess Log4j2 implementation smoothly. | | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | PingAccess doesn't support the following OpenTelemetry properties:- `otel.logs.exporter` - `otel.metrics.exporter` - `otel.instrumentation.netty.ssl.telemetry.enabled` - `otel.instrumentation.netty.connection-telemetry.enabled`While these properties are included in the file, they are disabled by default. It is not recommended to alter these values. | 3. Repeat these steps for each node in the cluster. 4. Start or restart PingAccess. ## Configure distributed tracing with environment variables To set your environment variables, use a deployment tool of your choice, such as [Kubernetes](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/) or [Docker Compose](https://docs.docker.com/compose/environment-variables/set-environment-variables/). ### Steps 1. To enable distributed tracing, set the environment variable `PA_RUN_PA_ENABLE_DISTRIBUTED_TRACING=true`. You can find more information about PingAccess environment variables in [Use environment variables to override configuration settings](../configuring_and_customizing_pingaccess/pa_environment_variables_config_override.html). 2. Review the default properties available for configuration in the default OpenTelemetry properties table. To convert an OpenTelemetry property to an environment variable: 1. Convert the name to uppercase. 2. Replace all dot (`.`) and hyphen (`-`) characters with underscores (`_`). #### Example: The `otel.exporter.otlp.endpoint` property is equivalent to the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | These environment variables are native to the OpenTelemetry library and are distinct from PingAccess's standard environment variable format (PA\_*\*\_*\*).Don't use the PingAccess "PA\_" prefix to set OpenTelemetry environment variables. Use the standard "OTEL\_" prefix as defined in the OpenTelemetry specification. | The following table lists the default properties in `opentelemetry.properties`. You can find the full list of configurable OpenTelemetry properties in the [OpenTelemetry SDK documentation](https://opentelemetry.io/docs/languages/java/configuration/). > **Collapse: Default OpenTelemetry properties** > > | Property | Description | > | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | > | `otel.service.name` (required) | Specify the name of your application or service. The default value is PingAccess. If running PingAccess in a clustered environment, it might be helpful to give each node a different otel.service.name so you can tell them apart if one is having latency issues. | > | `otel.exporter.otlp.endpoint` (required) | Specify the endpoint for your OpenTelemetry collector or backend:- If using OTLP or gRPC, use port 4317. > > For example, `http://localhost:4317`. > > - If using either OTLP or HTTP with protobuf, use port 4318.The default value is `http://localhost:4318`. | > | `otel.exporter.otlp.protocol` (required) | Specify the OTLP transport protocol to use for all telemetry data. Valid values include:- `http/protobuf` (default) > > PingAccess recommends using this setting for its simplicity and compatibility with firewalls. > > If using `http/protobuf`, make sure `otel.exporter.otlp.endpoint` is set to port 4318. > > - `grpc` to use OTLP or gRPC. > > If using `grpc`, make sure `otel.exporter.otlp.endpoint` is set to port 4317. > > - `http/json` to use either OTLP or HTTP with JSON. > > Review your backend OTEL consumer documentation for recommendations on which port to use for JSON. | > | `otel.resource.attributes` (optional) This property is commented out by default. | Optionally define static environment metadata for all telemetry data. Use the `key1=val,key2=val` format.For example, to add the environment, deployment zone, and host role, set `otel.resource.attributes=environment=dev,deployment.zone=us-east-1a,host.role=web-server`. | > | `otel.instrumentation.` (optional) This property is commented out by default. | You can use this property to disable specific instrumentation modules if they cause noise or conflicts. The value after otel.instrumentation must match the instrumentation name. For example, to disable JDBC instrumentation, set otel.instrumentation.jdbc.enabled=false. | > | `otel.javaagent.logging` (required) | You can use this property to control OpenTelemetry agent logging. The default value is `application`. Ping Identity recommends using the default value because it integrates with the PingAccess Log4j2 implementation smoothly. | | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | PingAccess doesn't support the following OpenTelemetry properties:- `otel.logs.exporter` - `otel.metrics.exporter` - `otel.instrumentation.netty.ssl.telemetry.enabled` - `otel.instrumentation.netty.connection-telemetry.enabled`While these properties are included in the file, they are disabled by default. It is not recommended to alter these values. | 3. Repeat these steps for each node in the cluster. 4. Start or restart PingAccess.