How to Create Install And Configure Multinode Apach Kafka Cluster

In Section You Will Learn How to Create Install And Configure Multinode Apach Kafka Cluster

Perform all steps into each and every servers


Pre-requisites

  1. Physical or Virtual Machine ideally with 4 GB RAM, 2 CPU cores and 20 GB disk space
  2. Linux operating system
  3. JDK 8

We will use three centos 7 servers for kafka configuration in GCP(Google cloud platform)

Installing Apache Kafka

centos7 servers into gcp dashborad

In this demo we are going to install - kafka v2.2 and Zookeeper v3.4

Use below link for kafka and zookeeper

  1. Download - kafka v2.2
  2. Download - Zookeeper v3.4

Update and install


 sudo -i
 yum update -y
 yum install wget -y


Do Network configuration

                         
10.128.0.29  kafka1.local.com kafka1
10.128.0.30  kafka2.local.com kafka2
10.128.0.31  kafka3.local.com kafka3
                
               

Disable Firewall or Add rule if you want

               
systemctl status firewalld.service
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl stop firewalld.service

                
               

Please make sure JDK 8 is installed

java-version

Create apache kafka home directory

               
mkdir -p /opt/apche_kafka
                
               

Download apache zookeeper tarball

               
wget http://mirrors.estointernet.in/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
                
               

Extract apache zookeeper tarball

               
tar -xvzf zookeeper-3.4.14.tar.gz

                
               

Move zookeeper extracted directory into apache kafka home

               
mv zookeeper-3.4.14 /opt/apche_kafka/

                
               
zookeeper directory list apache kafka

Download apache kafka tarball

               
wget http://mirrors.estointernet.in/apache/kafka/2.2.0/kafka_2.12-2.2.0.tgz
                
               

Extract apache kafka tarball

               
tar -xvzf kafka_2.12-2.2.0.tgz

                
               

Move apache kafka extracted directory into apache kafka home

               
mv kafka_2.12-2.2.0 /opt/apche_kafka/

                
               

Atlast, Change the ownership

               

chown -R root:root /opt/apche_kafka

                
               


Start Configuration:(Do it into all servers)

Zookeeper Configuration

Create zookeeper data directory

               

mkdir -p /opt/apche_kafka/zookeeper-3.4.14/zkdata

                
               

Goto zookpeer config directory

               

cd /opt/apche_kafka/zookeeper-3.4.14/conf


                
               

create zookeeper.properties file and copy paste below code and save it

               

vi zookeeper.properties

dataDir=/opt/apche_kafka/zookeeper-3.4.14/zkdata

clientPort=2181

maxClientCnxns=200
tickTime=2000

server.1=kafka1.local.com:2888:3888
server.2=kafka2.local.com:2888:3888
server.3=kafka3.local.com:2888:3888

initLimit=20
syncLimit=10



                
               

Create myid file for all servers 1,2,3 respectively and .cfg file

               

echo 1 > /opt/apche_kafka/zookeeper-3.4.14/myid
cp /opt/apche_kafka/zookeeper-3.4.14/conf/zoo_sample.cfg /opt/apche_kafka/zookeeper-3.4.14/conf/zoo.cfg

                
               

Kafka Configuration

Create kafka data directory

               

mkdir -p /opt/apche_kafka/kafka_2.12-2.2.0/kafka-logs

                
               

Goto kafka config directory

               

cd /opt/apche_kafka/kafka_2.12-2.2.0/config


                
               

Edit server.properties file update below parameters (broker id should be unique)

               

vi server.properties


##broker id should be unique for all brokers
broker.id=0. 


log.dirs=/opt/apche_kafka/kafka_2.12-2.2.0/kafka-logs


# Enter the zookeeper quorum details as below

zookeeper.connect=kafka1.local.com:2181,kafka2.local.com:2181,kafka3.local.com:2181



                
               

Start zookeeper into all servers and check status as well

               

nohup /opt/apche_kafka/zookeeper-3.4.14/bin/zkServer.sh start /opt/apche_kafka/zookeeper-3.4.14/conf/zookeeper.properties &

/opt/apche_kafka/zookeeper-3.4.14/bin/zkServer.sh status
                
               
check zookpeer status apache kafka

Start apache kafka into all servers

               

nohup /opt/apche_kafka/kafka_2.12-2.2.0/bin/kafka-server-start.sh  /opt/apche_kafka/kafka_2.12-2.2.0/config/server.properties &
                
               
start apache kafka broker
If You Want to Know More Checkout My Youtube Channel