Backup & Restore Kafka config

INTRODUCTION

Kafka is becoming the backbone of Event driven Architectures in many organisations. In the past all infrastructure configuration used to be stored and managed in the so called “CMDB”. That is not possible now with Kafka. Kafka being an event streaming platform, carries event data in Topics. Each topic has partitions. Depending on the size of the organisation the number of topics would range from about 100 to 5000. Security of Kafka topics includes Access controls.

Given the scale of the setup, it is crucial for organisations to keep a backup of the configuration of these topics and ACLs.

In this article we explain how you can backup and restore Kafka Topic configuration using Kafkawize.

TOPIC CONFIGURATION

A Kafka topic has lot of config parameters, the primary configuration of a topic includes, partitions, replication factor and sometimes retention period. The other configuration is not very relevant now for this post. Other aspect is who is authorised to access topic data. It is a producer or consumer access.

A Producer access includes host, principle and topic name. Access can also be on TransactionalID (for idempotent clients), and prefixed (default is Literal). Write and Describe access is provided.

A Consumer access includes host, principle, topic name, consumer group. Access can also be defined as Prefixed or Literal. (Read and Describe access is provided)

Principle based Access controls are better usually, as they are relying on certificates, and can be reused across different applications. However, there could be around 10k – 20k access controls in any company with growing Kafka usage. This configuration is stored in Zookeeper including Topic configuration.

Even though Zookeepers are running in a quorum, it is necessary to take back ups of the config. Let’s see how we can do that easily with Kafkawize.

Backup with Kafkawize

Kafkawize has a feature wherein it is possible to synchronise all the configuration of topics and ACLs in a cluster and store in a metastore like RDBMS or File. Kafkawize can read all topics and ACLs from cluster and show in the User Interface, for users to select few or all topics and synchronise to metastore.

You can associate each topic to a Topic owner and connect with Producer(s) teams and Consumer(s) teams in the Kafkawize metastore.

It doesn’t matter how many topics or ACLs exist in the cluster. All the config is now in Kafkawize in readable or easily exportable format.

Restore Config to Kafka Cluster

In case of an issue with the cluster, for instance a zookeeper crash or a data center failure, where all the topics and ACLs config is lost. In this situation, all the config from Kafkawize metastore can be restored back on the cluster with a single click. Kafkawize reads all topics and ACLs from metastore and show in the User Interface for users to select few or all topics and apply on the cluster.

Kafka cluster migration

Not only restore, when needed all the topics can be migrated from one cluster to another from Kafkawize. In the Kafkawize UI, you can select a source and target environment, and a push of a button will start to create topics and ACLs on the target cluster. You can select a few topics/ACLs or all the available topics/ACLs and start to synchronize.

Key Benefits

• Relying on one source of truth (kafkawize metastore)
• No risk of configuration loss
• Prevent manual creation of topics or ACLs on clusters
• Cluster to any cluster synchronisation
• Audit and logs of the entire activity

Conclusion

In this post we learnt the importance of taking backup of Topics and Access controls config, specially to handle unforeseen incidents. And how Kafkawize can easily and bring back Clusters to normal situation within an hour.