To give some background, Kafka topics are feeds where in messages are produced to and consumed from clients. It is crucial to define some naming conventions in the initial phases of the project, for proper Governance on the topics.
Topics have partitions and under the hood, there are files and folders created for every partition in the file system.
For example: if a topic name is “logstopic” and has 2 partitions (with one broker), there are folders in the file system (Kafka Broker logs directory configuration log.dirs) created for this topic like below.
If log.dirs=/usr/share/kafka, we should see 2 directories for this topic
/usr/share/kafka $ ls -ltr
Each of this directories contain files like below.
[kwuser@awsserver logstopic-0]$ ls -ltr
-rw-r–r–. 1 kwuser kwuser 3482 Oct 8 2019 00000000000000000000.log
-rw-r–r–. 1 kwuser kwuser 10485756 Jan 14 09:03 00000000000000000000.timeindex
-rw-r–r–. 1 kwuser kwuser 10485760 Jan 14 09:03 00000000000000000000.index
-rw-r–r–. 1 kwuser kwuser 8 Jan 14 09:04 leader-epoch-checkpoint
00000000000000000000.log file stores the actual events. More details on Log entries and segments : Kafka Log
Few limitations on how a topic name can be created
If you think any of the below don’t change in future, consider the below in the topic names.
Few examples : customernotifications, systemlog, webaudit, paymentack, analytics_post, mortgage_notifications_topic
Disable auto.create.topics.enable property on Kafka broker, so that users cannot create topics directly on Producing messages., and make sure someone or any application reviews the topic names before they getting created. Basically it has to enforce all the defined rules to follow certain guidelines. It is always better to automate this process and notify teams or Admin incase of violations.
It is not possible or supported by Kafka, to rename a Kafka topic.
As there are internal dependencies like consumer offset positions, offset checkpoint index etc, it is not an easy operation to rename. So as of now, only way to rename is, delete the topic and create another topic with a new name.
There is an issue raised to Kafka, but it is not going to be fixed. You may follow it. Jira Issue
However, to get back your data on the new topic, you can use Mirror Maker or Confluent replicator, and get data replicated.
Kafkawize allows users to request for topics with the above limitations mentioned, together with a Prefix configuration. In certain Kafka implementations, it is necessary to get topics created based on the environment.
Ex : devloggintopic, tstloggintopic, uatloggintopic, prdloggingtopic
If this has to be enforced, a Prefix can be configured per environment, and on requesting Kafka topics, it enforces users to apply the prefix, conforming to data integrity.
In this article, we learnt about how Kafka topics are created on the file system, and how certain naming conventions help in streamlining application events. As it is not possible to rename the topic, it’s better to get it the first time right.
Automate processes around your Kafka clusters with workflows, reduced risk full audit, governance.
© 2021 Kafkawize Technology Solutions. All rights reserved. Registered company in The Netherlands.