Kafka Installation and Execution

I have already shared the basics of Kafka, what is Kafka? What are its components in my previous article — Apache Kafka for Beginners.

Now let us know about the installations and usage in the applications:

Kafka Installations

• Before installation, let me tell me you that, we have different kind of kafka servers available in market.
• 1. Open Source : Apache Kafka
• 2. Commercial Distribution : Confluent Kafka
• 3. Managed Kafka Service : Confluent and AWS

In order to install that firstly visit https://kafka.apache.org/downloads

Now choose the version and click on link inside binary download.

• Usual steps to Run Kafka Server.

In order to run the Kafka Server

• Go to the folder where you have downloaded the kafka binary

Step 1: Start Zookeeper

• It will be present inside bin folder
• Also inorder to start zookeeper server, it requires zookeeper properties file to look into the information.
• So we have to provide the location of properties file also while starting zookeeper server.

.\bin\windows\zookeeper-server-start.bat config\zookeeper.properties

• here zookeeper will start with port 2181

Step 2: Start Kafka Server

• Again go to Kafka distribution. Inside bin go to windows folder.
• Now same thing will apply for Kafka also . While starting kafka server It will take config file.
• As Kafka itself a server the config file name is server.properties

.\bin\windows\kafka-server-start.bat config\server.properties.

It will take the default port to 9092

Step 3: Run the Kafka Topic to Create Topic

Before Creating Kafka Topic

• Now In a system there can be N number of kafka server/broker can be there . So while creating topic you should provide on which server you want to create Topic.
• Now we have to create the topic so this info also we have to provide.
• There can be N number of topic in the server . You can configure your topic for different purpose . It will be good if you Provide Topic Name here.
• Also how many partition on that topic you want. If you do not define it will take default partition of 1.
• Also, how many replication factor you want to keep. If you do not define by default 1.

.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --create --topic Test-Topic1 --partitions 3 --replication-factor 1

You can also get the all the list of created topics using

.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --list

You can also describe the topic for more information using

.\bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --describe --topic Test-Topic1

In order to monitor for bulk data, as data is being passed between different topics or not. All these monitoring can be done with the help of offset explorer tool, can be download from https://www.kafkatool.com/download.html

Now to produce and consume a message,

• Start the producer, but while creating or starting the producer we have to take care of
• There can be N number of brokers/servers will be there. Out of those list on which server you want to create the producer.
• Within that server from which topic you want to read the data.

Command to start Producer

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic Test-Topic2-topic

Now let us start Consumer

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic Test-Topic2-topic --from-beginning

Thank you for reading the article. Please clap, share and comment. it will encourage me to write more such articles. Do share your valuable suggestions, I appreciate your honest feedback!!!