PingOne Advanced Services

Monitoring and logging

With PingOne Advanced Services, your organization has its own dedicated cloud network that you can define, without having to manage cloud resources, containers, networking, scaling, healing, and backup and restoration.

Our Support team and the Site Reliability Engineers proactively monitor your infrastructure and deployments and attempt to address issues before they become problems. If outages occur, we’ll notify you using standard support methods.

You are responsible for monitoring your Ping applications and configurations, but we will stream log files to you, which will help you identify, monitor, and quickly resolve any issues you might encounter. You can also subscribe to receive alerts from PingOne Advanced Services, which will notify you of events occurring within your network.

See the following for details:

Platform monitoring

The PingOne Advanced Services platform is built on AWS, so we leverage their tools to monitor and support the platform.

These tools include:

  • Amazon CloudWatch, which is used for real-time monitoring of the AWS platform and our applications leveraging the platform.

  • GuardDuty, which is used for intelligent threat detection.

We also use New Relic, an observability platform used to log, track, and analyze data from any digital source in real time.

You are not responsible for monitoring the platform or its infrastructure, so you won’t see any of the alerts sent from these services, but rest assured that we will notify you if your network is affected.

You can also access the Ping Identity status page to see service interruption information, or subscribe to receive alerts regarding particular products or services.

Application monitoring and alerts

Since you are responsible for monitoring your applications, you can subscribe to receive a variety of different alerts regarding the health of your network. You can also work with your Ping Identity support teams to customize the alerts you receive and display them on Kibana dashboards.

Some of the most common PingFederate alerts and PingDirectory alerts are listed and described here.

PingFederate alerts

If more than one error occurs within one minute, alerts will be sent to those who subscribe to them. Alerts that PingFederate administrators often subscribe to include:

Error alerts Description

Fatal or critical errors

These errors were not discovered by other filters and likely require immediate attention.

Connectivity errors

These errors can occur for a variety of reasons and often indicate that cluster members cannot communicate with components, either within or outside of PingOne Advanced Services, or with each other.

PingFederate connectivity alerts are:

  • PF LDAP Connection Lost

  • PF PingID Connection Lost

Authentication flow errors

These errors indicate that the authentication flow contains errors, which are often HandleAuthNRequest errors.

PingFederate authentication flow alerts are:

  • PF Unexpected Runtime Error

  • PF Auth Exception

Invalid action errors

These errors indicate that a large number of unexpected invalid calls or actions were detected.

The PingFederate invalid action error is:

  • PF Invalid Request Parameter

Decoding errors

These errors indicate that a large number of token decoding errors were detected.

The PingFederate decoding error is:

  • PF Profile Message Missing ID

PingDirectory alerts

If more than one error occurs within one minute, alerts will be sent to those who subscribe to them. Alerts that PingDirectory administrators often subscribe to include:

Error alerts Description

Fatal or critical errors

These errors were not discovered by other filters and likely require immediate attention.

The PingDirectory critical error is:

  • PD Critical

General errors

These errors indicate that issues within the application might negatively affect the user experience.

PingDirectory general alerts are:

  • PD Major

  • PD Third Party Extension Exception

Connectivity errors

These errors can occur for a variety of reasons and often indicate that cluster members cannot communicate with components, either within or outside of PingOne Advanced Services, or with each other.

The PingDirectory connectivity error is:

  • PD LDAP Connection Handler Startup Error

Replication errors

These errors indicate that data consistency issues exist.

PingDirectory replication alerts are:

  • PD Failed Mirror Configuration

  • PD Replication Backlogged

  • PD Replication Changelog Failure

  • PD Replication Missing Changes

  • PD Replication Replay Operation Failed

  • PD Replication Server Failure Alarm

  • PD Unresolved Replication Conflict

Performance errors

These errors indicate that performance has fluctuated more than it normally does.

The PingDirectory performance alert is:

  • PD Worker Threads Terminated

Garbage collection errors

These errors indicate that garbage collection processes are occurring more frequently, or are taking longer than expected.

The PingDirectory garbage collection filter alert is:

  • PD Continuous Garbage Collection Filter

Log streaming

Event logs contain valuable information regarding possible security threats, outages, and metrics that can help troubleshoot issues, and log streaming makes it possible to export these log files in near real-time.

You can stream your log files for all Ping Identity products in your PingOne Advanced Services cloud network using a variety of different log aggregation tools. Either set up this process when you initially access your applications, or submit a service request at any time. Ensure that you include pertinent information, listed here, regarding your aggregation tool in your request.

See the following for details:

Amazon S3 Bucket

To export log files with an Amazon S3 bucket, include the following information in your request:

  • Your AccountID

  • AWS key and secret

  • Bucket name

Amazon CloudWatch

To export log files with an Amazon CloudWatch, include the following information in your request:

  • Your AccountID

  • AWS key and secret

  • Log group names

  • Log steam names

Elasticsearch

You can configure Elasticsearch to use the Elasticsearch output plugin.

Generic HTTP or Webhook

To export log files with a generic HTTP or webhooks, include the following information in your request:

  • Endpoint URL

  • HTTP method used to send data

  • An authorization token or key (optional)

IBM QRadar

With QRadar, the syslog output with a JSON encoding is used. To export log files, include the destination host and port in your request.

Microsoft Azure

You can configure Azure for one of these plugins:

Splunk HTTP Event Collector (HEC)

With Splunk HEC, only RAW Endpoint is supported. To export log files, include the following information in your request:

  • Splunk HEC Endpoint URL

  • Splunk API Key

Syslog

To export log files with Syslog, include the destination host and port in your request.

Product-specific logs

Detailed information regarding product log files is available in the product documentation:

PingFederate:

PingDirectory:

PingAccess:

Updating your log-streaming services

In the July 2022 release, PingOne Advanced Services added a configurable log-streaming pipeline, which made it possible to customize the ways log data is streamed. You can filter streamed data by application, log, and keywords, and modify JSON files.

If you have not yet migrated to the new streaming processes, be aware that streaming formatting changes need to occur and are outlined here. This format is based on logstash output filters and can be customized to meet your needs.

Here is an example of a log event at the input:

{"@timestamp":"2022-07-14T12:00:10.728763Z",
"message":"All pods in namespace ingress-nginx-public are running              |
PASS |\n",
"log_type":"customer_out",
"time":"2022-07-14T12:00:04.314969694Z",
"kubernetes":
{"container_hash":"public.ecr.aws/r2h3l6e4/pingcloud-services/robot-framework@sh
a256:e64b3beb9c23d655f8542e685f0c68c01178498f4b226294f36773832dd1cb48",
"container_image":"public.ecr.aws/r2h3l6e4/pingcloud-services/robot-framework:v1
.3.0",
"docker_id":"9944534ac7556566a40a9f40331e5d07b0cf4244cca2a5a07e6f4b83d0de69a9",
"labels":
{
"app":"ping-cloud",
"controller-uid":"ea007809-075f-4cd6-9955-ad69c94ae190",
"job-name":"healthcheck-cluster-health-27630000"
},
"container_name":"healthcheck-cluster-health",
"host":"ip-10-254-1-222.us-west-2.compute.internal",
"pod_id":"05dc5c68-852e-40f9-93a8-13a24502d545",
"namespace_name":"ping-cloud-antonklyba",
"pod_name":"healthcheck-cluster-health-27630000-4fztv"},
"host":"10.254.12.248",
"@version":"1",
"stream":"stdout",
"log_group":"application"
}

This file contains the following information:

  • @timestamp: Indicates when logstash processed the event.

  • log or message: Ping Identity applications generate logs and all other types of applications and sidecars.

  • log_type: Provides internal labels for all events sent to the pipeline.

  • time: Date and time when the log was captured, which could be different from the date and time it was generated.

  • kubernetes: The nested JSON object with kubernetes metadata.

  • host: The internal IP address of a fluent-bit pod sent to logstash.

  • @version: This internal logstash field will always be “1”.

  • stream: The name of the stream where the log was captured, and will either be stdout (standard output) or stderr (standard error).

  • log_group: This internal label will always be “application”.

If you do not apply filters and opt to use the default output configuration, a variety of differences exist:

  • If you are exporting log files with Amazon CloudWatch or generic HTTP, you will receive a JSON file similar to the example.

  • If you are exporting log files to an Amazon S3 Bucket:

    • Ensure that the S3 output is appropriately configured. Use ‘codec ⇒ “json”’ to create a JSON file similar to the example. The S3 output is useless when filters are not applied or this setting is not established because it only obtains @timestamp, host, and message fields from the event.

    • By default, the following line will be written to text files named 'ls.s3.${randomUUID}.${currentTime}.${tags}.${part}.txt`

      ${timestamp} ${host} ${message}(e.g “2022-07-14T12:00:04.314969694Z 10.254.12.248 All pods in namespace ingress-nginx-private are running | PASS |”)
      Filenames within S3 buckets can be prefixed to create directories, but the filenames are hardcoded and cannot be changed.

  • If you are exporting log files to Syslog, it uses the rfc3164 format, by default:

    ${timestamp} ${host} ${process}: ${message} (e.g. “Jul 14 12:00:04 10.254.12.248 LOGSTASH[-]: All pods in namespace ingress-nginx-private are running               |PASS |”)

    Ensure that Syslog output is appropriately configured. Either use ‘codec ⇒ “json”’ to send the whole JSON object in a ${message} field, or configure the message property to include specific fields.