Understanding Data Engineering and Its Workflow
Follow sample transactions from their source to a table ready for analysis.
Understanding data, one pipeline at a time.
Hello, I’m the writer behind alfianh.id. Here I collect notes on how data moves from its source to something useful for making decisions.
I work through small examples: transactions that need cleaning, transformations that need tests, and jobs that must be safe to repeat. Start with foundations, then examine design choices and their failure modes.
Author identity is a preview. Articles and data on this site are learning samples.
Find a learning pathStart with the flow
An editorial selection: understand the concept, try SQL, then think about recovery.
SQL for Cleaning Transaction Data
Normalize missing identifiers, choose a deterministic duplicate winner, and reconcile rejected transactions.
Designing Idempotent and Reliable Pipelines
Use a business key and source revision to make a destination update safe to replay.
Follow your question
9 published articles in English
Latest writing
In publication order. Different reads from the editorial selection above.
Browse the archiveTechnical rendering sample
A small sample for checking technical prose and bilingual publishing.
Understanding Event Streaming with Kafka
Reason about partition order, consumer groups, offsets, and duplicate delivery using a small event log.
Tested Data Transformations with dbt
Connect a model with ref, test its business key, and review incremental correctness separately.
Learn in sequence
Beginner
Data engineering foundations
Follow data from its source through cleaning and ETL to an analytical model.
4 ordered articles
Be comfortable reading tables. For the Python exercise, have Python 3 and basic knowledge of functions.
Practitioner · Intermediate to advanced
Reliable pipelines
Connect scheduling, transformation tests, and safe replay.
3 ordered articles
Finish Foundations or be comfortable with SQL, Python functions, ETL, and dimensional models.
A design to inspect
Illustrative case study: synthetic transactions, quality checks, quarantine, and batch recovery. No client or business impact claims.
Explore the synthetic transaction pipeline