도커 컨테이너로 Kafka, ZooKeeper 그리고 Kafdrop을 구성하는 방법

도커 컨테이너로 Kafka, ZooKeeper 그리고 Kafdrop을 구성하는 방법

구성 요소

Kafka : 메시지 브로커 역할

ZooKeeper : Kafka 클러스터 메타데이터 저장소

Kafdrop : Kafka 메시지 확인용 웹 UI 대시보드

 

작업 디렉토리 생성

mkdir -p /docker-container/docker-kafka

cd /docker-container/docker-kafka

docker compose 파일 작성

vim docker-compose.yml

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.1
    container_name: zookeeper
    hostname: zookeeper
    environment:
      TZ: Asia/Seoul
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - zookeeper_data:/var/lib/zookeeper/data
      - /etc/localtime:/etc/localtime:ro
    networks:
      - kafka-net
    healthcheck:
      #      test: ["CMD-SHELL", "echo ruok | nc -w 2 localhost 2181 | grep imok"]
      test: ["CMD-SHELL", "timeout 3 bash -c '</dev/tcp/127.0.0.1/2181'"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  kafka:
    image: confluentinc/cp-kafka:7.6.1
    container_name: kafka
    hostname: kafka
    depends_on:
      - zookeeper
    environment:
      TZ: Asia/Seoul
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_MESSAGE_TIMESTAMP_TYPE: CreateTime
    volumes:
      - kafka_data:/var/lib/kafka/data
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "9092:9092"
      - "29092:29092"
    networks:
      - kafka-net
    healthcheck:
      test: ["CMD-SHELL", "echo > /dev/tcp/localhost/9092 || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  kafdrop:
    image: obsidiandynamics/kafdrop:4.0.1
    container_name: kafdrop
    hostname: kafdrop
    depends_on:
      - kafka
    environment:
      TZ: Asia/Seoul
      JVM_OPTS: -Duser.timezone=Asia/Seoul
      KAFKA_BROKERCONNECT: kafka:9092
    volumes:
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "9999:9000"
    networks:
      - kafka-net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/actuator/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

networks:
  kafka-net:
    name: kafka-net
    driver: bridge

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

컨테이너 실행

docker compose up -d

docker compose ps --format "table {{.Name}}\t{{.Service}}\t{{.Status}}\t{{.Ports}}"

NAME        SERVICE     STATUS                    PORTS
kafdrop     kafdrop     Up 45 minutes (healthy)   0.0.0.0:9999->9000/tcp, [::]:9999->9000/tcp
kafka       kafka       Up 45 minutes (healthy)   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp
zookeeper   zookeeper   Up 45 minutes (healthy)   2181/tcp, 2888/tcp, 3888/tcp

728x90

Kafka 테스트

토픽 생성

docker compose exec kafka kafka-topics \
  --create \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 \
  --partitions 1 \
  --topic my-topic

Created topic my-topic.

토픽 목록 확인

docker compose exec kafka kafka-topics \
  --list \
  --bootstrap-server localhost:9092

my-topic

메시지 송수신

메시지 프로듀서 실행(메시지 전송)

docker compose exec kafka kafka-console-producer \
  --broker-list localhost:9092 \
  --topic my-topic

메시지 입력

>Hello Kafka
>첫 번째 메시지
>안녕하세요!

• Ctrl + C로 종료

메시지 컨슈머 실행(메시지 수신)

docker compose exec kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic my-topic \
  --from-beginning

Hello Kafka
첫 번째 메시지
안녕하세요!

• Ctrl + C로 종료

Kafdrop 접속

브라우저에서 접속

• Kafdrop UI를 통해 토픽, 메시지, 파티션 등을 확인할 수 있습니다.

http://localhost:9999

kafdrop

 

Kafka 브로커, ZooKeeper, Kafdrop을 도커로 구성하고 메시지를 송수신할 수 있는 환경이 준비되었습니다.