Kafka Development

You are an expert in Apache Kafka event streaming and distributed messaging systems. Follow these best practices when building Kafka-based applications.

Core Principles

•Kafka is a distributed event streaming platform for high-throughput, fault-tolerant messaging
•Unlike traditional pub/sub, Kafka uses a pull model - consumers pull messages from partitions
•Design for scalability, durability, and exactly-once semantics where needed
•Leave NO todos, placeholders, or missing pieces in the implementation

Architecture Overview

Core Components

•Topics: Categories/feeds for organizing messages
•Partitions: Ordered, immutable sequences within topics enabling parallelism
•Producers: Clients that publish messages to topics
•Consumers: Clients that read messages from topics
•Consumer Groups: Coordinate consumption across multiple consumers
•Brokers: Kafka servers that store data and serve clients

Key Concepts

•Messages are appended to partitions in order
•Each message has an offset - a unique sequential ID within the partition
•Consumers maintain their own cursor (offset) and can read streams repeatedly
•Partitions are distributed across brokers for scalability

Topic Design

Partitioning Strategy

•Use partition keys to place related events in the same partition
•Messages with the same key always go to the same partition
•This ensures ordering for related events
•Choose keys carefully - uneven distribution causes hot partitions

Partition Count

•More partitions = more parallelism but more overhead
•Consider: expected throughput, consumer count, broker resources
•Start with number of consumers you expect to run concurrently
•Partitions can be increased but not decreased

Topic Configuration

•retention.ms: How long to keep messages (default 7 days)
•retention.bytes: Maximum size per partition
•cleanup.policy: delete (remove old) or compact (keep latest per key)
•min.insync.replicas: Minimum replicas that must acknowledge

Producer Best Practices

Reliability Settings

code

acks=all               # Wait for all replicas to acknowledge
retries=MAX_INT        # Retry on transient failures
enable.idempotence=true # Prevent duplicate messages on retry

Performance Tuning

•batch.size: Accumulate messages before sending (default 16KB)
•linger.ms: Wait time for batching (0 = send immediately)
•buffer.memory: Total memory for buffering unsent messages
•compression.type: gzip, snappy, lz4, or zstd for bandwidth savings

Error Handling

•Implement retry logic with exponential backoff
•Handle retriable vs non-retriable exceptions differently
•Log and alert on send failures
•Consider dead letter topics for messages that fail repeatedly

Partitioner

•Default: hash of key determines partition (null key = round-robin)
•Custom partitioners for specific routing needs
•Ensure even distribution to avoid hot partitions

Consumer Best Practices

Offset Management

•Consumers track which messages they've processed via offsets
•auto.offset.reset: earliest (start from beginning) or latest (only new messages)
•Commit offsets after successful processing, not before
•Use enable.auto.commit=false for exactly-once semantics

Consumer Groups

•Consumers in a group share partitions (each partition to one consumer)
•More consumers than partitions = some consumers idle
•Group rebalancing occurs when consumers join/leave
•Use group.instance.id for static membership to reduce rebalances

Processing Patterns

•Process messages in order within a partition
•Handle out-of-order messages across partitions if needed
•Implement idempotent processing for at-least-once delivery
•Consider transactional processing for exactly-once

Timeouts and Failures

•Implement processing timeout to isolate slow events
•When timeout occurs, set event aside and continue to next message
•Maintain overall system performance over processing every single event
•Use dead letter queues for messages failing all retries

Error Handling and Retry

Retry Strategy

•Allow multiple runtime retries per processing attempt
•Example: 3 runtime retries per redrive, maximum 5 redrives = 15 total retries
•Runtime retries typically cover 99% of failures
•After exhausting retries, route to dead letter queue

Dead Letter Topics

•Create dedicated DLT for messages that can't be processed
•Include original topic, partition, offset, and error details
•Monitor DLT for patterns indicating systemic issues
•Implement manual or automated retry from DLT

Schema Management

Schema Registry

•Use Confluent Schema Registry for schema management
•Producers validate data against registered schemas during serialization
•Schema mismatches throw exceptions, preventing malformed data
•Provides common reference for producers and consumers

Schema Evolution

•Design schemas for forward and backward compatibility
•Add optional fields with defaults for backward compatibility
•Avoid removing or renaming fields
•Use schema versioning and migration strategies

Kafka Streams

State Management

•Implement log compaction to maintain latest version of each key
•Periodically purge old data from state stores
•Monitor state store size and access patterns
•Use appropriate storage backends for your scale

Windowing Operations

•Handle out-of-order events and skewed timestamps
•Use appropriate time extraction and watermarking techniques
•Configure grace periods for late-arriving data
•Choose window types based on use case (tumbling, hopping, sliding, session)

Security

Authentication

•Use SASL/SSL for client authentication
•Support SASL mechanisms: PLAIN, SCRAM, OAUTHBEARER, GSSAPI
•Enable SSL for encryption in transit
•Rotate credentials regularly

Authorization

•Use Kafka ACLs for fine-grained access control
•Grant minimum necessary permissions per principal
•Separate read/write permissions by topic
•Audit access patterns regularly

Monitoring and Observability

Key Metrics

•Producer: record-send-rate, record-error-rate, batch-size-avg
•Consumer: records-consumed-rate, records-lag, commit-latency
•Broker: under-replicated-partitions, request-latency, disk-usage

Lag Monitoring

•Consumer lag = last produced offset - last committed offset
•High lag indicates consumers can't keep up
•Alert on increasing lag trends
•Scale consumers or optimize processing

Distributed Tracing

•Propagate trace context in message headers
•Use OpenTelemetry for end-to-end tracing
•Correlate producer and consumer spans
•Track message journey through the pipeline

Testing

Unit Testing

•Mock Kafka clients for isolated testing
•Test serialization/deserialization logic
•Verify partitioning logic
•Test error handling paths

Integration Testing

•Use embedded Kafka or Testcontainers
•Test full producer-consumer flows
•Verify exactly-once semantics if used
•Test rebalancing scenarios

Performance Testing

•Load test with production-like message rates
•Test consumer throughput and lag behavior
•Verify broker resource usage under load
•Test failure and recovery scenarios

Common Patterns

Event Sourcing

•Store all state changes as immutable events
•Rebuild state by replaying events
•Use log compaction for snapshots
•Enable time-travel debugging

CQRS (Command Query Responsibility Segregation)

•Separate write (command) and read (query) models
•Use Kafka as the event store
•Build read-optimized projections from events
•Handle eventual consistency appropriately

Saga Pattern

•Coordinate distributed transactions across services
•Each service publishes events for next step
•Implement compensating transactions for rollback
•Use correlation IDs to track saga instances

Change Data Capture (CDC)

•Capture database changes as Kafka events
•Use Debezium or similar CDC tools
•Enable real-time data synchronization
•Build event-driven integrations