Blog

  • Interactive Kafka Monitoring Tutorial

    Interactive Kafka Monitoring Tutorial

    Interactive Kafka Monitoring Tutorial

    An interactive guide to understanding real-time data streams with Kafka. Explore the concepts, examine the code, and see a live simulation.

    Introduction

    This tutorial will guide you through setting up a simple Python script to monitor real-time events flowing through your Kafka broker, specifically focusing on player actions from your MUD game. This is a fundamental step in understanding how data streams work in a distributed system. The interactive elements below allow you to explore the core concepts, review the Python code, and even simulate a live event stream to solidify your understanding. If you haven’t already, check out my first post on building an interactive MUD game with Confluent here Confluent and Python MUD Game Tutorial.

    Key Concepts

    These are the core Kafka concepts involved in real-time monitoring. Each concept is expanded below for your easy reference.

    The `monitor_events.py` Script

    This is the Python script you’ll create to connect to your Kafka broker, subscribe to the player event topic, and print messages to the console. Review the code and feel free to copy it directly.

    Code: `monitor_events.py`

    # monitor_events.py

from confluent_kafka import Consumer, KafkaException
import json
import sys

KAFKA_BOOTSTRAP_SERVERS = 'localhost:9092'
PLAYER_EVENTS_TOPIC = 'player_events'

consumer_conf = {
    'bootstrap.servers': KAFKA_BOOTSTRAP_SERVERS,
    'group.id': 'monitor_group',
    'auto.offset.reset': 'earliest'
}

def run_monitor():
    print("Starting Kafka event monitor...")
    consumer = None
    try:
        consumer = Consumer(consumer_conf)
        consumer.subscribe([PLAYER_EVENTS_TOPIC])

        print(f"Subscribed to topic: {PLAYER_EVENTS_TOPIC}...")

        while True:
            msg = consumer.poll(timeout=1.0)
            if msg is None: continue

            if msg.error():
                if msg.error().code() != KafkaError._PARTITION_EOF:
                    raise KafkaException(msg.error())
            else:
                event_data = json.loads(msg.value().decode('utf-8'))
                print(f"EVENT: {json.dumps(event_data)}")

    except KeyboardInterrupt:
        print("Monitor shutting down.")
    finally:
        if consumer:
            consumer.close()

if __name__ == "__main__":
    run_monitor()

    Setup Checklist

    • ✓ Kafka Broker Active (e.g., Docker)
    • ✓ `game_engine.py` Service Running
    • ✓ `confluent-kafka` Library Installed (`pip install confluent-kafka`)

    Live Event Stream (Simulation)

    Observe how events flow in real-time! Click ‘Start’ to simulate player actions being published to the `player_events` topic and watch them appear here as if a Kafka consumer was reading them.

    > Press ‘Start Simulation’ to begin monitoring the ‘player_events’ topic…

  • Confluent and Python MUD Game Tutorial

    Introduction

    This tutorial presents a decoupled architecture for a real-time interactive game. It leverages Apache Kafka for asynchronous communication between a Telnet server and a game engine. Player location and inventory are persisted in a MySQL database, allowing players to leave and return to their last saved state. The project is an excellent exercise in building distributed, real-time systems.

    System Architecture

    The application is split into two main services: a Telnet Server that handles raw client connections and a Game Engine that processes all game logic. These services communicate asynchronously using Kafka as a message bus, which makes the system scalable and resilient. Player state is stored externally in a MySQL database. Get the source code here if you want to build it – https://github.com/pattonjava/confluentmudgame.

    Telnet Client

    Player’s Terminal
    ↔

    Telnet Server

    Manages Connections
    ↔

    Apache Kafka

    → Commands Topic →

    ← Events Topic ←

    ↔

    Game Engine

    Processes Logic

    ↔

    MySQL DB

    Persists Player State

    Prerequisites

    Before you begin, ensure the following software is installed and running on your Linux server. This guide assumes a basic familiarity with the Linux command line and system administration.

    • Python 3.8+: Your primary programming language.
    • pip: The Python package installer.
    • Java Runtime Environment (JRE): A dependency for running Kafka & Zookeeper.
    • Apache Kafka & Zookeeper: The message broker at the core of our system.
    • MySQL Server: The database for storing persistent player data.
    • Python Libraries: Install the required libraries with pip. 
      pip install confluent-kafka mysql-connector-python

    System Setup

    Follow these steps to configure the necessary infrastructure for the game: Apache Kafka for messaging and MySQL for data storage. All commands should be run from your Linux server’s terminal.

    1. Kafka Setup

    1. Download and Extract Kafka: 
      wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
    2. Start Zookeeper: In a separate terminal, run: 
      bin/zookeeper-server-start.sh config/zookeeper.properties
    3. Start Kafka Broker: In another terminal, run: 
      bin/kafka-server-start.sh config/server.properties
    4. Create Game Topics: Create the two topics the game requires. 
      bin/kafka-topics.sh --create --topic player_commands --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic game_events --bootstrap-server localhost:9092

    2. MySQL Setup

    1. Connect to MySQL: 
      mysql -u your_mysql_user -p
    2. Create Database and Table: Execute this SQL to create the `players` table. 
      CREATE DATABASE IF NOT EXISTS mud_game;
USE mud_game;

CREATE TABLE IF NOT EXISTS players (
    player_name VARCHAR(255) PRIMARY KEY,
    current_room VARCHAR(255) NOT NULL DEFAULT 'start_room',
    inventory JSON,
    health INT NOT NULL DEFAULT 100,
    last_login TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

    Code Walkthrough: File Overviews

    This project’s logic is distributed across several Python files. Understanding the role of each file is key to grasping the overall architecture.

    `config.py` centralizes all configuration settings for the game, including Kafka broker addresses, topic names, and MySQL database connection details. This file ensures that sensitive information and adjustable parameters are kept separate from the core logic, making the application easier to configure and deploy across different environments.

    `telnet_server.py` acts as the frontend for the MUD. It’s responsible for handling incoming Telnet client connections, managing input/output for each player (in separate threads), and serving as the bridge to Kafka. It publishes raw player commands to a Kafka topic and subscribes to another Kafka topic to receive and relay game events back to the respective players.

    `game_engine.py` is the brain of the MUD. It consumes player commands from Kafka, processes all game logic (e.g., player movement, item interaction, combat, chat), and updates the game state. Critically, it interacts with the MySQL database to load a player’s previous location and inventory upon connection and persists their state whenever significant changes occur (like moving to a new room or picking up an item). It then publishes relevant game events back to Kafka for the Telnet server to distribute to players.

    The `game_data/` directory contains files that define the static elements of the game world:

    • `game_data/items.py`: Defines the `Item` class and a dictionary of all static item definitions (e.g., ‘dusty map’, ‘rusty key’).
    • `game_data/npcs.py`: Defines the `NPC` (Non-Player Character) class and a dictionary of all static NPC definitions.
    • `game_data/rooms.py`: Defines the `Room` class and contains the `load_world_data` function, which constructs the game’s map (WORLD_MAP) by linking rooms and populating them with instances of items and NPCs based on their definitions.

    Running the Game

    With the infrastructure running and code in place, you can now start the game services and connect as a player. It’s recommended to run each service as a `systemd` daemon for proper management.

    1. Start Infrastructure: Ensure Zookeeper, Kafka, and MySQL are running.
    2. Start the Game Engine: Run the `game_engine.py` script. For production, create a systemd service file for it. 
      # Example systemd command
sudo systemctl start game_engine.service
    3. Start the Telnet Server: Run the `telnet_server.py` script, also preferably as a service. 
      # Example systemd command
sudo systemctl start telnet_server.service
    4. Connect as a Player: Use a Telnet client like PuTTY to connect to your server’s IP address on port 2323.

    Exploring Kafka Concepts

    This project uses several key Kafka features. Understanding them will help you build more complex and resilient real-time applications. Click on each concept to learn more.