Contents
This is sample teaching content. Publication dates and transaction records are illustrative.
A batch asks which records belong in a bounded run. A stream keeps arriving, so a consumer needs a durable position and a rule for retrying work. This introduction uses Kafka 4.1 concepts without a particular client library. It is a topic starting point, not a complete streaming curriculum.
Read a partitioned log
partition 0: offset 40 order-A-created
partition 0: offset 41 order-A-paid
partition 1: offset 12 order-B-created
Offsets 40 and 41 establish order inside partition 0. Offset 12 in partition 1 cannot be compared with them to infer global event order. When related events need partition-local ordering, producers must route them consistently, often using the same key. Changing partition counts or a partitioning strategy requires reviewing that assumption.
Within a consumer group, each assigned partition has one consumer at a time. Another group can independently read the same topic. Adding consumers beyond the partition count does not create extra partition parallelism for that group.
Choose when to commit
A committed offset identifies where the group should resume; it denotes the next record to read. After processing offset 41, commit 42 for that partition. If the consumer writes the destination and crashes before that commit, the record may be delivered again.
read offset 41
write event to destination using event_id
commit next offset 42
This sequence supports at-least-once handling when failures and commits are managed correctly. The destination must tolerate replay. Committing first creates the opposite risk: a crash before the destination write can lose that effect. Kafka transactions can coordinate supported Kafka reads and writes, but an arbitrary external database does not automatically share that atomic boundary.
Design a replay exercise
Send two events for one order. Stop the consumer after the destination write but before offset commit, then restart it. Check that event_id uniqueness prevents duplicate effects and that a late update cannot overwrite a newer revision. Monitor lag per partition as well as processing errors; a healthy loop can still produce incorrect business data.
Retention limits how far a consumer can replay. Agree on recovery requirements before relying on the log as a permanent archive. Consult the Kafka design reference for the guarantees of your chosen deployment and client.