Backend Software Engineer - Kafka (Core & Internal Distribution)

Job Description

About the Role

We are building an in-house team to maintain and contribute to our internal Apache Kafka distribution. This is not an operations role - you will work directly on the Kafka codebase itself: i.e. the Broker, KRaft controller, protocol layer, clients, and storage engine.

Working alongside senior technical anchors, you will help maintain our internal Kafka distribution, backport critical fixes, contribute upstream to Apache Kafka, and build benchmarking and testing pipelines for a large-scale production fleet.

We already do this successfully with GlusterFS, where we maintain an in-house distribution backed by deep upstream expertise, and we are replicating that model for Kafka.

Key Responsibilities

• Build and maintain performance testing suites, benchmarks, and validation tools for local patches.
• Maintain our internal Kafka distribution: patch integration, bug fixes, and release testing.
• Contribute upstream to Apache Kafka through test-cases, doc-reviews, bug-reports, bug fixes, code reviews, and community design discussions (mid to long term).
• Diagnose and fix broker and controller issues related to replication, partition handling, KRaft consensus, and storage management.
• Collaborate with platform engineering teams to advise on broker behavior, configuration, and capacity decisions.
• Document internal patches, architectural changes, and upgrade procedures.

What Are We Looking For?

• Codebase & Systems Familiarity: Strong background in Java or Scala with experience reading, modifying, or debugging JVM-based system code.
• Distributed Systems Fundamentals: Understanding of consensus protocols, replication models, log-structured storage, and distributed state management.
• Systems Programming: Familiarity with JVM internals (garbage collection, memory layout) and operating system I/O (page cache, zero-copy, socket tuning).
• Development Experience: Several years of demonstrable experience in systems or infrastructure software engineering, working with distributed middleware or database engines.

Good to Have

• Prior contributions to Apache Kafka or comparable distributed systems (e.g., Pulsar, BookKeeper, ZooKeeper, etcd).
• Familiarity with KRaft consensus or ZooKeeper-less Kafka architecture.
• Experience with Kafka Streams or Kafka Connect internals.
• Experience using Kafka system tests, Trogdor, or duct tape test frameworks.

What This Role Is NOT

• Not an SRE/Operations Role: Dedicated platform teams operate the clusters; you will write and fix the software they run.
• Not Application Development: You will work directly inside Kafka rather than building consumer applications.
• Not a Consulting Role: You will write code, review patches, and ship production-ready software.

What We Offer

• Dedicated focus and time for open-source upstream contributions.
• Opportunity to validate code against a massive production fleet and observe large-scale distributed behavior.
• Direct mentorship from senior systems engineers and technical anchors.
• Proven distribution model and strong organizational backing for core open-source development.