Designing Idempotent and Reliable Pipelines
Use a business key and source revision to make a destination update safe to replay.
Reliability
Notes, examples, and design choices for this topic.
Open this topic in the archive · All topics
Filters and display
3 articles · Reliability · All levels · Newest · Page 1/1
Tested Data Transformations with dbt
Connect a model with ref, test its business key, and review incremental correctness separately.
Scheduling Pipelines with Apache Airflow
Define task dependencies and bounded retries while keeping batch publication safe to repeat.