Contents
A sample shop records transactions in its checkout application. The team wants daily sales totals, but incoming files can contain duplicate rows or missing dates. Data engineering prepares a repeatable path from those records to a place where people can analyze them.
From source to result
The source produces transaction records. An ingest step copies them into raw storage. A transformation then checks and arranges the data into a useful table. That table can feed a report; it is not the report itself.
Keep a raw copy before changing it. If cleaning rules change, the copy lets the team compare a new result with the original input. Limit access and retention according to the needs of the actual data.
One example check
The following query calculates totals from transactions with both an identifier and an amount:
SELECT transaction_date, SUM(amount) AS total_amount
FROM transactions
WHERE transaction_id IS NOT NULL
AND amount IS NOT NULL
GROUP BY transaction_date;
This query does not handle duplicate transactions or missing dates. Count excluded rows and investigate their cause before using the result for a decision. Replacing missing values with zero without understanding them can hide a source problem.
Terms to distinguish
| Term | Meaning in this example |
|---|---|
| Ingest | Copy transactions from the source system. |
| Transformation | Check and arrange transactions for the next use. |
| Pipeline | A sequence of steps that moves and transforms data. |
Start with a question you can check, such as whether the raw row count matches the accepted and rejected counts. Then decide who needs to know when that check fails.