KAFKA Admin

··      Responsibilities
·      Design, implement, and maintain highly available Kafka clusters in production environments.
·      Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
·      Administer security protocols and access controls to protect the platform.
·      Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
·      Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
·      Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
·      Assist with planning and managing cluster resources to support future growth.
·      Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
·      Manage a tune of Kafka Connect, Kafka Streams, and Mirror Maker as needed for data pipelines.
·      Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
·      Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).
·      Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
·      Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
·      Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
·      Perform root cause analysis and resolution of platform-level incidents and outages.
·      Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.