Getting Started with Kafka Producer and Consumer in C#: A Simple Guide

Build Your First Kafka Producer and Consumer in C# Step-by-Step

Apache Kafka is a distributed event-streaming platform designed for high-throughput, low-latency handling of real-time data. It’s widely used for building applications that need to process and exchange data efficiently. In this guide, we’ll walk through building a simple Kafka Producer and Consumer using C#.

---

What is Kafka?

Kafka works on the publish-subscribe model:

• Producer: Sends messages (events) to a Kafka topic.

• Consumer: Reads messages from a Kafka topic.

Kafka ensures durability, scalability, and fault tolerance. Messages in Kafka are organized into topics, and each topic can be split into partitions for parallel processing.

---

Prerequisites

1. Kafka Installed: Install Kafka locally or use a cloud-hosted Kafka service like Confluent Cloud.

2. C# Development Environment: Use an IDE like Visual Studio or Visual Studio Code.

3. NuGet Package: Install the Confluent.Kafka library to interact with Kafka.

Step 1: Install the Required Library

Add the Confluent.Kafka package to your project. Run the following command in the terminal:

dotnet add package Confluent.Kafka

Step 2: Configure Kafka

Kafka Broker

Kafka requires a broker to connect Producers and Consumers. If Kafka is running locally, the default broker is usually localhost:9092.

Topic

Create a Kafka topic named demo-topic. If you’re using a local Kafka setup, use the following command in the terminal:

kafka-topics.sh --create --topic demo-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Step 3: Kafka Producer in C#

The Producer sends messages to a Kafka topic.

using Confluent.Kafka;
using System;
using System.Threading.Tasks;

class KafkaProducer
{
    public static async Task ProduceMessages(string broker, string topic)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = broker
        };

        using var producer = new ProducerBuilder<Null, string>(config).Build();

        Console.WriteLine("Enter messages to send to Kafka (type 'exit' to quit):");

        while (true)
        {
            var message = Console.ReadLine();

            if (message.ToLower() == "exit")
                break;

            try
            {
                var deliveryResult = await producer.ProduceAsync(topic, new Message<Null, string> { Value = message });
                Console.WriteLine($"Message '{message}' sent to Partition: {deliveryResult.Partition}, Offset: {deliveryResult.Offset}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error sending message: {ex.Message}");
            }
        }
    }

    static async Task Main(string[] args)
    {
        await ProduceMessages("localhost:9092", "demo-topic");
    }
}

Explanation

1. ProducerConfig: Specifies the Kafka broker (localhost:9092 in this example).

2. ProduceAsync: Sends a message to the Kafka topic.

3. Topic: All messages are sent to the demo-topic.

---

Step 4: Kafka Consumer in C#

The Consumer reads messages from a Kafka topic.

using Confluent.Kafka;
using System;
using System.Threading;

class KafkaConsumer
{
    public static void ConsumeMessages(string broker, string topic, string groupId)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = broker,
            GroupId = groupId,
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();

        consumer.Subscribe(topic);

        Console.WriteLine($"Listening to topic '{topic}'...");

        try
        {
            while (true)
            {
                var consumeResult = consumer.Consume(CancellationToken.None);
                Console.WriteLine($"Message received: {consumeResult.Message.Value}");
            }
        }
        catch (OperationCanceledException)
        {
            Console.WriteLine("Consumption cancelled.");
        }
        finally
        {
            consumer.Close();
        }
    }

    static void Main(string[] args)
    {
        ConsumeMessages("localhost:9092", "demo-topic", "demo-group");
    }
}

Explanation

1. ConsumerConfig: Defines settings like the Kafka broker, group ID, and auto-offset behavior.

2. Subscribe: Subscribes the consumer to the topic demo-topic.

3. Consume: Listens for new messages and processes them.

Step 5: Test the Producer and Consumer

1. Start Kafka: Make sure your Kafka broker is running.

2. Run the Consumer:

   • Start the KafkaConsumer application.

   • The consumer will wait for messages from the topic demo-topic.

3. Run the Producer:

   • Start the KafkaProducer application.

   • Type messages into the console, and press Enter to send them to the topic.

You should see the messages sent by the producer appear in the consumer's console output.

Output

Producer Console

Enter messages to send to Kafka (type 'exit' to quit):
Hello, Kafka!
Message 'Hello, Kafka!' sent to Partition: 0, Offset: 0

Consumer Console

Listening to topic 'demo-topic'...
Message received: Hello, Kafka!

Explaining Kafka to a Layman

Think of Kafka like a mailing service:

1. Producer: The sender who writes a letter (message) and posts it to a mailbox (topic).

2. Consumer: The receiver who checks the mailbox and reads the letter.

Kafka ensures that the message (letter) is safely stored in the mailbox until the receiver (consumer) picks it up.

Benefits of Kafka

1. Scalable: Kafka can handle millions of messages per second.

2. Durable: Messages are stored on disk, ensuring reliability.

3. Decoupled Systems: Producers and Consumers don’t need to know about each other, making the architecture modular.

---

Conclusion

Congratulations! You’ve successfully built a simple Kafka producer and consumer in C#. While this guide focused on the basics, Kafka offers advanced features like partitioning, replication, and stream processing, which are worth exploring as you scale your applications.

Have questions or ideas about Kafka? Drop a comment below!