Deleting Kafka Topics on Docker
In this post we’re going to learn how to delete a Kafka Topic when running a Kafka Broker on Docker.
We’ll spin up a local Kafka environment using the Docker Compose template from the Kafka Basic Tutorial blog post that I wrote last week. Let’s open a terminal window and run the following commands to set up our environment:
git clone firstname.lastname@example.org:mneedham/basic-kafka-tutorial.git && cd basic-kafka-tutorial docker-compose up
On another terminal window, run the following command to see the list of Docker containers that are running:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3851185ae46f jupyter/scipy-notebook:latest "tini -g -- start-no…" 7 days ago Up 3 minutes 0.0.0.0:8888->8888/tcp jupyter-tutorial a118a60010ce confluentinc/cp-enterprise-kafka "/etc/confluent/dock…" 7 days ago Up 3 minutes 0.0.0.0:9092->9092/tcp, 9093/tcp broker-tutorial 1606bfe6e93d confluentinc/cp-zookeeper "/etc/confluent/dock…" 7 days ago Up 4 minutes 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper-tutorial
We’re going to connect to
broker-tutorial, seen on the 2nd line of this output, which is the container for our Kafka broker.
We’ll use the
docker exec command to connect to the broker, and use the
kafka-topics command to create and delete a topic.
Jacek Laskowski gives a detailed explanation of this command in The Internals of Apache Kafka gitbook.
Let’s start by creating a dummy topic, which we can do with the following command:
$ docker exec broker-tutorial kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic blog-dummy Created topic "blog-dummy".
So far, so good.
Note that we also to pass in the
--zookeeper argument to tell the command where our Zookeeper Instance is running.
zookeeper:2181 value is derived from the
hostname:ZOOKEEPER_CLIENT_PORT values from the following fragment of our Docker Compose file:
zookeeper: image: confluentinc/cp-zookeeper hostname: zookeeper container_name: zookeeper-tutorial ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000
We can list all the topics available on this broker by running the following command:
$ docker exec broker-tutorial kafka-topics --list --zookeeper zookeeper:2181 __confluent.support.metrics _confluent-metrics blog-dummy
There are a couple of Confluent metrics topics, but we can see our
blog-dummy topic as well.
Now let’s delete it by running the following command:
$ docker exec broker-tutorial kafka-topics --delete --zookeeper zookeeper:2181 --topic blog-dummy Topic blog-dummy is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.
We didn’t actually set the
delete.topic.enable parameter, so maybe our topic isn’t actually going to be deleted.
If we look back at the other terminal window, however, we’ll see the following output:
broker-tutorial | [2019-05-23 06:29:23,307] INFO [Topic Deletion Manager 1], Handling deletion for topics blog-dummy (kafka.controller.TopicDeletionManager) broker-tutorial | [2019-05-23 06:29:23,308] INFO [Topic Deletion Manager 1], Deletion of topic blog-dummy (re)started (kafka.controller.TopicDeletionManager) broker-tutorial | [2019-05-23 06:29:23,308] INFO [Topic Deletion Manager 1], Topic deletion callback for blog-dummy (kafka.controller.TopicDeletionManager) broker-tutorial | [2019-05-23 06:29:23,348] INFO [Topic Deletion Manager 1], Deletion of topic blog-dummy successfully completed (kafka.controller.TopicDeletionManager)
So it seems like our topic was deleted.
And, in fact, scrolling back through the output from running
docker-compose up we’ll find the following line:
broker-tutorial | delegation.token.max.lifetime.ms = 604800000 broker-tutorial | delete.records.purgatory.purge.interval.requests = 1 broker-tutorial | delete.topic.enable = true broker-tutorial | fetch.purgatory.purge.interval.requests = 1000 broker-tutorial | group.initial.rebalance.delay.ms = 0
So it seems like this property is set by default in the version that we’re using. We can list the topics one more time just to be sure:
$ docker exec broker-tutorial kafka-topics --list --zookeeper zookeeper:2181 __confluent.support.metrics _confluent-metrics
The Confluent Metrics topics remain, but
blog-dummy is nowhere to be seen.
About the author
I'm currently working on real-time user-facing analytics with Apache Pinot at StarTree. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also I co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.