PingIntelligence

Change Kafka default settings

Kafka and Zookeeper will be installed as part of the deployment framework.

By default, in the config/hosts file, Kafka and Zookeeper are configured to be installed as part of the deployment framework:

# this option can be used if there is an existing kafka installation that can be used
# set it to false if kafka need not be installed
install_kafka=true
  • To disable the Kafka and Zookeeper installation, set install_kafka=false.

  • Edit the parameters in config/kafka-defaults.yml to change the default installation settings for Kafka and Zookeeper.

    kafka:
     # Define ports for the Kafka brokers
     # These ports remain same for all brokers
     ssl_port: 9094
     #Port to be used for Communication with abs and dataengine
     sasl_port: 9093
    
     #kafka jks password
     jks_password: changeme
    
    
     #Enable Delete topics in kafka
     delete_topic: false
    
     ssl_key_alias: pingidentity
     wait_time_before_clean: 30
     startup_timeout: 120
    
    zookeeper:
    
       # Define ports for the zookeeper brokers
       # These ports remain same for all zookeeper
       ssl_port: 2182
    Kafka variables
    Variable Description

    ssl_port

    SSL port for Kafka. default: 9094

    sasl_port

    SASL port that is also used by the data engine and ABS to communicate with Kafka default: 9093

    jks_password

    JKS password. If a custom truststore and keystore is provided, you can configure the password here.

    delete_topic

    Enables topic deletion in Kafka.

    wait_time_before_clean

    Waiting time before cleaning existing data in Kafka, for new installation.

    startup_timeout

    Waiting time for Kafka to start.

    Zookeeper variables
    Variable Description

    ssl_port

    SSL port for Zookeeper, also used for communication to Kafka via Kafka’s SSL port.. default :9093

Changing default Kafka configurations in config/hosts

The config/hosts file has a Kafka configuration, for example:

[kafka]
172.16.40.81 zookeeper_id=1

You can install Kafka and Zookeeper as a cluster by providing multiple IPs or hosts in the host file, and zookeeper_id according to the number of nodes to install.

  • zookeeper_id should start from 1.

  • Each new node’s zookeeper_id should increase by 1.

  • The nodes should be listed in ascending zookeeper_id order.

  • The number of nodes should only be either one or three, for example:

    • One node:

      [kafka]
      <IP address> zookeeper_id=1
    • Three nodes:

      [kafka]
      <IP address 1> zookeeper_id=1
      <IP address 2> zookeeper_id=2
      <IP address 3> zookeeper_id=3

You can provide custom crt and key files in the certs folder location:

kafka_private_key_location: "{{ cert_dir }}/kafka/kafka.key"
kafka_cert_location: "{{ cert_dir }}/kafka/kafka.crt"

Configure Kafka keystore password

You can configure the keystore password in config/kafka-defaults.yml. The keystore and truststore will be generated dynamically and will be used for zookeeper-kafka and kafka-client communication.