About this session
Kafka, stream-processing engines, and Apache Doris each maintain their own progress and transaction state. This talk follows one Kafka event from a partition, through ingestion, to a visible Doris row - and explains the coordination that makes this path reliable.
We trace three designs: Kafka -> Doris, Kafka -> Flink -> Doris, and Kafka -> Spark -> Doris. In Routine Load, Doris reads Kafka records and saves its place in the topic together with the data it loads. In Flink, the engine saves its Kafka progress, asks Doris to prepare the data for commit, and finalizes the Doris write only when the checkpoint succeeds. In Spark, each micro-batch prepares its Doris writes, then the driver finalizes them for that batch. Finally, Doris makes the committed data available for queries.
After following the normal flow, we examine what happens if a worker, the processing engine, or Doris fails along the way. In every path, the system must remember both where to resume reading from Kafka and which Doris write belongs to that progress. On restart, it must safely finish the same write or replay the data without creating duplicates. We also clarify the difference between data that is prepared, committed, and finally visible to queries.
Attendees will learn how Kafka offsets/Share group, checkpoints or epochs, Doris Stream Load 2PC, transaction labels, FE metadata, and publish-to-visible work together - so they can choose the right ingestion path, validate no-data-loss behavior, and debug duplicates or recovery failures. The talk will also explore how Kafka Share Groups are expected to evolve to support higher-throughput consumption in the future.
