❯Delta Lake Exactly-Once Guarantees and Idempotent Pipelines in Spark.

In this article, I examine Spark’s strategy for achieving exactly-once delivery, with a particular focus on the mechanisms within Structured Streaming. The foundation of these guarantees lies in the use of checkpoints which store the metadata necessary to define each micro-batch and track the progress of data transfers, and on the guarantees offered by the sink, which play a fundamental role.

To fix the ideas consider streaming data from a Kafka topic (e.g. orders):

stream = (
    spark.readStream
         .format("kafka")
         .option("subscribe", "orders")
         .load()
)
 
query = (
    stream.writeStream
          .format("parquet")
          .option("checkpointLocation", "/checkpoints/orders")
          .start("/output")
)

Here we're writing our checkpoints to /checkpoints/orders and the output to /output.

From a high-level perspective the checkpoint is similar in spirit to a write-ahead log for the engine. In practice the checkpoint is a directory with several subdirectories:

• The state/ subdirectory contains information regarding the data that is being tracked for stateful operations. If the streaming query is completely stateless (e.g., a simple select or filter), this directory will not be created at all. However, for stateful queries (aggregations, joins, etc.), this directory is where Spark stores its running state across micro-batches using a state store provider (like HDFS or RocksDB).

• The sources/ directory keeps track of the lineage and identity of your input sources. It ensures that the query remains consistent across restarts. It stores source-specific metadata required for recovery.

• The offsets/ directory is where the Spark driver records the start and end positions of the data to be processed from the source (e.g., Kafka offsets like {"orders":{"0":14230}}) before a micro-batch begins. That's how the batch can be retried in case of failure. Reading the last entry in this directory together with the last commit information, allows Spark to know where to resume the job making sure it does not skip data.

• The commits/ directory contains files (empty or with simple metadata) that prove the engine finished processing the batch successfully. If the job fails processing a batch before a corresponding commit file is written, Spark knows it has to reprocess data starting from where the last offsets/ indicates.

Essentially the offsets/ and commits/ directories are the basis for the exactly-once guarantee:

Input Discovery in Spark: Ingestion and Checkpoint Update Cycle

Spark is not continuously scanning files or storage locations, nor it's constantly scanning Kafka topics. A high level list of the main input update mechanisms is the following:

1. Directory listing mode (default)

   • Periodically lists new objects in cloud storage
   • Uses incremental listing (based on internal markers, not full scans)
   • New files → added to internal “discovered but not processed” queue

2. Kafka consumer

   • Pull based
   • Periodic polls for offsets and partitions
   • Possibility to enable continuous/long lived consumption

In any case now that we have a better idea of ingestion we can move on to discuss the fetch + process + update cycle. From a high-level perspective it goes like this:

This cycle guarantees that

1. if the job crashes after writing the data but before the checkpoint, Spark will replay that batch after restart.
2. If the job crashes after the checkpoint, the batch is not retried but skipped (Spark knows it has been already successfully processed and committed).

This means that spark provides deterministic replay: a batch can be reprocessed more than once, with Spark knowing what input records belong to that batch.

How Exactly-Once Semantics is Achieved in Spark

Consider the case of a Spark crash after the data is committed to the sink but before recording the batch commit. When the driver retries the batch, the data is processed again and written to the sink.

The checkpoint metadata is not sufficient to guarantee exactly-once semantics, the fundamental missing piece is the guarantees and behavior of the sink.

In other words, Spark Structured Streaming's end-to-end exactly-once guarantee is really a combination of Spark's execution guarantees and the sink's write semantics. For example:

File Sink: If a batch is retried, Spark simply writes a new set of files alongside the old ones. The standard file sink achieves exactly-once because it writes a separate manifest log inside the output directory (/output/_spark_metadata). A downstream reader reads that log to know which files are valid. The orphaned files from the failed attempt are left cluttering your storage forever (unless manually cleaned), they are never overwritten or deleted by Spark. Notice how duplication happens, but a downstream reader knows which files are valid and avoids reading twice.

Delta Lake: It uses a mechanism called idempotent writes via transaction identifiers. When Spark writes to Delta, it passes a unique Application ID (txnAppId) and the Micro-batch ID (a transaction version) (txnVersion) . When Delta attempts to commit a new JSON file to its _delta_log, it checks if an entry with that exact txnAppId and txnVersion already exists in its transaction history. If it does, Delta recognizes it as a duplicate execution of a previously completed batch and silently ignores the write, ensuring no data is duplicated.

Kafka Sink: Spark uses Kafka's transactional producer logic (must be configured) to guarantee exactly-once delivery.

Counter Example: Writing to a Database Suppose Spark writes

INSERT INTO orders ...

1000 rows are inserted. Then Spark crashes before updating its checkpoint. After restart the INSERT is performed again and the same 1000 rows are inserted creating duplicates.

Unless your table has, a primary key, a unique constraint, or you're doing an UPSERT/MERGE, you'll get duplicates.

In this case, Spark has replayed the same batch correctly, but the sink cannot distinguish a retry from a new write.

Conclusion

The result can be summarized as follows:

Spark guarantees deterministic replay of micro-batches. To achieve end-to-end exactly-once results, the sink must provide semantics that make replay safe, either through idempotent writes, transactional commits, or another mechanism that detects and ignores duplicate batch executions.

Idempotency is one way to achieve this, but it's not the only one. Transactional sinks (like Delta Lake or Kafka with transactions) achieve exactly-once without requiring every individual write operation to be idempotent.

So Spark does ultimately rely on the sink's capabilities for end-to-end exactly-once guarantees. This logic is on the same line of the one I described in previous article about idempotent pipelines with BigQuery .

Spark can ensure it processes the same input batch consistently and only advances its checkpoint when appropriate, but it cannot, by itself, prevent duplicate effects in an external system that has no notion of retries or transactions.