PingIntelligence

Changing Kafka and Zookeeper default settings

Kafka and Zookeeper will be installed as part of the deployment framework.

By default, in the config/hosts file, Kafka and Zookeeper are configured to be installed as part of the deployment framework:

# this option can be used if there is an existing kafka installation that can be used
# set it to false if kafka need not be installed
install_kafka=true

  • To disable the Kafka and Zookeeper installation, set install_kafka=false.

  • Edit the parameters in config/kafka-defaults.yml to change the default installation settings for Kafka and Zookeeper.

    kafka:
 # Define ports for the Kafka brokers
 # These ports remain same for all brokers
 ssl_port: 9094
 #Port to be used for Communication with abs and dataengine
 sasl_port: 9093

 #kafka jks password
 jks_password: changeme


 #Enable Delete topics in kafka
 delete_topic: false

 ssl_key_alias: pingidentity
 wait_time_before_clean: 30
 startup_timeout: 120

zookeeper:

   # Define ports for the zookeeper brokers
   # These ports remain same for all zookeeper
   ssl_port: 2182
    Kafka variables
    Variable Description

    ssl_port

    Secure Sockets Layer (SSL) port for Kafka.

    Default: 9094

    sasl_port

    Simple Authentication and Security Layer (SASL) port that is also used by the data engine and ABS to communicate with Kafka

    Default: 9093

    jks_password

    Java KeyStore (JKS) password.

    If a custom truststore and key store is provided, you can configure the password here.

    delete_topic

    Enables topic deletion in Kafka.

    wait_time_before_clean

    Waiting time before cleaning existing data in Kafka, for new installation.

    startup_timeout

    Waiting time for Kafka to start.
    Zookeeper variables
    Variable Description

    ssl_port

    SSL port for Zookeeper, also used for communication to Kafka via Kafka’s SSL port.

    Default: 9093

Changing default Kafka configurations in config/hosts

The config/hosts file has a Kafka configuration, for example:

[kafka]
172.16.40.81 zookeeper_id=1

You can install Kafka and Zookeeper as a cluster by providing multiple IPs or hosts in the host file, and zookeeper_id according to the number of nodes to install.

  • zookeeper_id should start from 1.

  • Each new node’s zookeeper_id should increase by 1.

  • The nodes should be listed in ascending zookeeper_id order.

  • The number of nodes should only be either one or three, for example:

    • One node:

      [kafka]
<IP address> zookeeper_id=1

    • Three nodes:

      [kafka]
<IP address 1> zookeeper_id=1
<IP address 2> zookeeper_id=2
<IP address 3> zookeeper_id=3

You can provide custom crt and key files in the certs folder location:

kafka_private_key_location: "{{ cert_dir }}/kafka/kafka.key"
kafka_cert_location: "{{ cert_dir }}/kafka/kafka.crt"

Configure Kafka keystore password

You can configure the keystore password in config/kafka-defaults.yml. The keystore and truststore will be generated dynamically and will be used for zookeeper-kafka and kafka-client communication.