PingAccess

Distributed tracing

Distributed tracing provides visibility into the full path a request takes in a distributed system, helping to instrument, collect, and export telemetry data. Telemetry data can make it easier to identify the root cause of performance issues or errors, especially when investigating backchannel calls.

Ping Identity supports the OpenTelemetry framework (OTEL) for collecting distributed tracing data. PingAccess uses OpenTelemetry Protocol (OTLP) to send distributed traces to a backend service such as Jaeger for collection, storage, and visualization.

Distributed tracing is an evolving feature in PingAccess. It’s subject to change without notice, even in a minor or maintenance release.

What is distributed tracing?

In a distributed system, requests pass through multiple services hosted on multiple servers. Distributed tracing shows you how an incoming request was processed across all servers and services, including:

  • Which servers and services the request went through.

  • How much time each service took to process its part of the request.

  • How the services are connected.

  • What the failure point was in case of a request failure.

Supported request types

PingAccess supports distributed tracing for the following request types:

  • Incoming HTTP requests

  • Outgoing HTTP requests

  • Internal Java database connectivity (JDBC) requests

Distributed tracing components

Traces

A trace represents the path of a request through an application. A trace is made up of one or more spans. Learn more about traces in the OpenTelemetry documentation.

Spans

A span is a segment of a request journey. It represents a unit of work or an operation within a service. Each span includes the following elements:

  • traceId represents the trace that the span is a part of.

  • spanId is a unique ID for the span.

  • parentSpanId is the ID of the originating request.

Root span

The root span indicates the start and end of an entire operation. The parentSpanId of the root span is null because the root span isn’t part of an existing trace. Subsequent spans in the trace have their own unique spanId. Their traceId is the same as that of the root span, and their parentSpanId matches the spanId of the root span.

OpenTelemetry

OpenTelemetry is an open-source observability framework for instrumenting, generating, collecting, and exporting telemetry data. It provides a standardized way to capture distributed traces across different services and platforms. It doesn’t provide a backend for storing or analyzing telemetry data. Learn more in the OpenTelemetry documentation.

Visualizing traces

PingAccess can push traces to an OpenTelemetry Protocol (OTLP) endpoint over HTTP. Any backend that supports OTLP and HTTP can be used to collect and visualize the traces.

Try the Jaeger tracing All-in-one Docker image to capture exported spans. By default, Jaeger stores the spans in memory, but you can configure Jaeger to send the spans to various persistent datastores external to the Docker image.

PingAccess includes an exchangeId tag on each operation it performs. You can use this tag to compare a specific span with a corresponding entry in the PingAccess log. Debug log level entries can provide additional insight after you use a tool like Jaeger to identify where the problem might lie.

Configuring distributed tracing

Enable distributed tracing and configure relevant OpenTelemetry settings using one of the following methods:

  • Edit the run.properties and opentelemetry.properties files directly.

  • Use environment variables to set the properties in the run.properties and opentelemetry.properties files.

    Properties set by environment variables take precedence over any property values specified in the opentelemetry.properties file.

If you’re running PingAccess as a Windows service, you must reinstall the service for changes to the pa.enable.distributed.tracing property to take effect. This limitation applies regardless of the configuration method you use.

  • With the property files

  • With environment variables

Configure distributed tracing with property files

Steps

  1. Enable distributed tracing:

    1. Open the <PA_HOME>/conf/run.properties file in a text editor.

    2. Set the pa.enable.distributed.tracing property to true.

      Example:

      pa.enable.distributed.tracing=true

  2. To configure OpenTelemetry properties, open the <PA_HOME>/conf/opentelemetry.properties file in a text editor.

    The following table lists the default properties in opentelemetry.properties. You can find the full list of configurable OpenTelemetry properties in the OpenTelemetry SDK documentation.
    Default OpenTelemetry properties
    Property Description

    otel.service.name (required)

    Specify the name of your application or service. The default value is PingAccess.

    If running PingAccess in a clustered environment, it might be helpful to give each node a different otel.service.name so you can tell them apart if one is having latency issues.

    otel.exporter.otlp.endpoint (required)

    Specify the endpoint for your OpenTelemetry collector or backend:

    • If using OTLP or gRPC, use port 4317.

      For example, http://localhost:4317.

    • If using either OTLP or HTTP with protobuf, use port 4318.

    The default value is http://localhost:4318.

    otel.exporter.otlp.protocol (required)

    Specify the OTLP transport protocol to use for all telemetry data. Valid values include:

    • http/protobuf (default)

      PingAccess recommends using this setting for its simplicity and compatibility with firewalls.

      If using http/protobuf, make sure otel.exporter.otlp.endpoint is set to port 4318.

    • grpc to use OTLP or gRPC.

      If using grpc, make sure otel.exporter.otlp.endpoint is set to port 4317.

    • http/json to use either OTLP or HTTP with JSON.

      Review your backend OTEL consumer documentation for recommendations on which port to use for JSON.

    otel.resource.attributes (optional)

    This property is commented out by default.

    Optionally define static environment metadata for all telemetry data. Use the key1=val,key2=val format.

    For example, to add the environment, deployment zone, and host role, set otel.resource.attributes=environment=dev,deployment.zone=us-east-1a,host.role=web-server.

    otel.instrumentation.<instrumentation_name> (optional)

    This property is commented out by default.

    You can use this property to disable specific instrumentation modules if they cause noise or conflicts.

    The value after otel.instrumentation must match the instrumentation name.

    For example, to disable JDBC instrumentation, set otel.instrumentation.jdbc.enabled=false.

    otel.javaagent.logging (required)

    You can use this property to control OpenTelemetry agent logging. The default value is application.

    Ping Identity recommends using the default value because it integrates with the PingAccess Log4j2 implementation smoothly.

    PingAccess doesn’t support the following OpenTelemetry properties:

    • otel.logs.exporter

    • otel.metrics.exporter

    • otel.instrumentation.netty.ssl.telemetry.enabled

    • otel.instrumentation.netty.connection-telemetry.enabled

    While these properties are included in the file, they are disabled by default. It is not recommended to alter these values.

  3. Repeat these steps for each node in the cluster.

  4. Start or restart PingAccess.

Configure distributed tracing with environment variables

To set your environment variables, use a deployment tool of your choice, such as Kubernetes or Docker Compose.

Steps

  1. To enable distributed tracing, set the environment variable PA_RUN_PA_ENABLE_DISTRIBUTED_TRACING=true.

    You can find more information about PingAccess environment variables in Use environment variables to override configuration settings.

  2. Review the default properties available for configuration in the default OpenTelemetry properties table. To convert an OpenTelemetry property to an environment variable:

    1. Convert the name to uppercase.

    2. Replace all dot (.) and hyphen (-) characters with underscores (_).

      Example:

      The otel.exporter.otlp.endpoint property is equivalent to the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

      These environment variables are native to the OpenTelemetry library and are distinct from PingAccess’s standard environment variable format (PA_<FILENAME>_<PROPERTY>).

      Don’t use the PingAccess "PA_" prefix to set OpenTelemetry environment variables. Use the standard "OTEL_" prefix as defined in the OpenTelemetry specification.

      The following table lists the default properties in opentelemetry.properties. You can find the full list of configurable OpenTelemetry properties in the OpenTelemetry SDK documentation.

      Default OpenTelemetry properties
      Property Description

      otel.service.name (required)

      Specify the name of your application or service. The default value is PingAccess.

      If running PingAccess in a clustered environment, it might be helpful to give each node a different otel.service.name so you can tell them apart if one is having latency issues.

      otel.exporter.otlp.endpoint (required)

      Specify the endpoint for your OpenTelemetry collector or backend:

      • If using OTLP or gRPC, use port 4317.

        For example, http://localhost:4317.

      • If using either OTLP or HTTP with protobuf, use port 4318.

      The default value is http://localhost:4318.

      otel.exporter.otlp.protocol (required)

      Specify the OTLP transport protocol to use for all telemetry data. Valid values include:

      • http/protobuf (default)

        PingAccess recommends using this setting for its simplicity and compatibility with firewalls.

        If using http/protobuf, make sure otel.exporter.otlp.endpoint is set to port 4318.

      • grpc to use OTLP or gRPC.

        If using grpc, make sure otel.exporter.otlp.endpoint is set to port 4317.

      • http/json to use either OTLP or HTTP with JSON.

        Review your backend OTEL consumer documentation for recommendations on which port to use for JSON.

      otel.resource.attributes (optional)

      This property is commented out by default.

      Optionally define static environment metadata for all telemetry data. Use the key1=val,key2=val format.

      For example, to add the environment, deployment zone, and host role, set otel.resource.attributes=environment=dev,deployment.zone=us-east-1a,host.role=web-server.

      otel.instrumentation.<instrumentation_name> (optional)

      This property is commented out by default.

      You can use this property to disable specific instrumentation modules if they cause noise or conflicts.

      The value after otel.instrumentation must match the instrumentation name.

      For example, to disable JDBC instrumentation, set otel.instrumentation.jdbc.enabled=false.

      otel.javaagent.logging (required)

      You can use this property to control OpenTelemetry agent logging. The default value is application.

      Ping Identity recommends using the default value because it integrates with the PingAccess Log4j2 implementation smoothly.

      PingAccess doesn’t support the following OpenTelemetry properties:

      • otel.logs.exporter

      • otel.metrics.exporter

      • otel.instrumentation.netty.ssl.telemetry.enabled

      • otel.instrumentation.netty.connection-telemetry.enabled

      While these properties are included in the file, they are disabled by default. It is not recommended to alter these values.

  3. Repeat these steps for each node in the cluster.

  4. Start or restart PingAccess.