Data Quality: Testing, Data Contracts and Observability for Pipelines

Nothing destroys trust in a data team faster than a wrong number in a board deck. Once stakeholders catch one bad metric, they start double-checking everything, and the platform loses its value. Data quality isn't a one-off cleanup project. It's an engineering practice built into every pipeline.

#Key takeaways

• Data quality has several dimensions: accuracy, completeness, consistency, timeliness, uniqueness and validity.
• Test data like code: schema, not-null, uniqueness, relationships, accepted values, and business rules.
• Data contracts make producer–consumer expectations explicit and stop upstream changes from silently breaking pipelines.
• Observability (freshness, volume, distribution and schema monitoring) catches problems that you didn't write tests for.
• Clear ownership and incident processes matter as much as tooling.

#The dimensions of data quality

Different datasets need different emphasis. Finance needs accuracy and completeness; a real-time operations dashboard needs freshness.

#Layer 1: Tests in the pipeline

Write assertions that run every time data is built. With SQL-based transformation tools like dbt (dbt explained), common tests are declarative:

yamlmodels:
  - name: fct_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: status
        tests:
          - accepted_values:
              values: [pending, paid, shipped, cancelled, refunded]

Add business-rule tests written as SQL that returns failing rows:

sql-- Test fails if any order's line items don't sum to its subtotal
SELECT o.order_id
FROM fct_orders o
JOIN (SELECT order_id, SUM(amount) AS items_total
      FROM fct_order_lines GROUP BY order_id) l USING (order_id)
WHERE ABS(o.subtotal - l.items_total) > 0.01;

#What to test first

1. Primary keys: unique and not null on every important model.
2. Foreign keys: relationships between facts and dimensions (data modeling with star schemas).
3. Critical business rules behind key metrics (revenue, active users).
4. Accepted values for status and enum fields.
5. Freshness of sources feeding executive dashboards.

#Severity levels

Not every failure should stop the pipeline:

• Error (block): duplicate primary keys in finance tables. Stop and alert.
• Warn: a small percentage of nulls in an optional field. Alert and continue.

#Layer 2: Data contracts

Many data incidents start upstream: an application engineer renames a column, changes a field's meaning or stops populating it. The application works fine, but downstream analytics silently break.

A data contract is an explicit agreement between a data producer and its consumers:

yamlcontract: orders.v1
owner: checkout-team
schema:
  order_id:     {type: string, required: true, unique: true}
  customer_id:  {type: string, required: true}
  total_cents:  {type: integer, required: true, minimum: 0}
  currency:     {type: string, enum: [USD, EUR, GBP]}
  created_at:   {type: timestamp, required: true}
guarantees:
  freshness: "events available within 15 minutes"
  compatibility: "backward compatible; breaking changes require new version + 30 days notice"

How contracts are enforced:

• Schema validation in CI for the producing service, so breaking changes fail the build.
• Schema registries for event streams (Apache Kafka explained).
• Validation at ingestion, rejecting or quarantining non-conforming records.

Contracts shift quality left, to the moment a change is made, instead of discovering breakage days later in a dashboard.

#Layer 3: Data observability

Tests catch the problems you anticipated. Observability catches the ones you didn't, by monitoring data behavior over time:

• Freshness: when was the table last updated? Alert if it's late.
• Volume: row counts per load compared to history. A 60% drop usually means an upstream failure.
• Distribution: null rates, value ranges, category proportions. For example, a sudden spike in country = NULL.
• Schema changes: columns added, removed or retyped.
• Lineage: which downstream models and dashboards depend on a broken table, so you know the blast radius.

Simple volume and freshness monitors on your most important tables provide a lot of protection for little effort.

#Layer 4: Handling bad data

When data fails validation, you have options:

• Fail the pipeline: safest for critical data; no bad data reaches consumers.
• Quarantine: route bad records to a separate table for review, and load the good ones.
• Repair automatically: deduplicate, standardize formats, fill defaults, but only for well-understood cases.
• Flag and publish: mark affected metrics with a warning banner in dashboards.

Whichever you choose, never fail silently.

#Incident management for data

Treat data incidents like production incidents (incident management and postmortems):

1. Detect: tests, monitors or a stakeholder report.
2. Communicate: tell affected consumers quickly, before they present the wrong number.
3. Contain: pause downstream syncs (reverse ETL, ML training) if bad data would spread.
4. Fix and backfill: correct the logic, re-run affected intervals (idempotent pipelines make this safe: Airflow best practices).
5. Review: add a test or monitor so the same failure is caught automatically next time.

#Ownership

Tooling fails without clear responsibility:

• Every source and critical model has a named owning team.
• Certified datasets are marked, documented and tested to a higher bar; ad-hoc tables aren't.
• Metric definitions live in one place (a semantic layer or documented models), not in every dashboard.

#A 30-day data quality plan

• Week 1: Identify the 10 most important tables and metrics. Add primary key and not-null tests.
• Week 2: Add freshness and volume monitors with alerts to owners.
• Week 3: Add business-rule tests for top metrics. Document definitions.
• Week 4: Introduce a data contract for the most incident-prone source. Run a retro on recent incidents and convert lessons into tests.

#FAQ

#What's the difference between data quality testing and data observability?

Testing checks explicit rules you define (unique keys, valid values). Observability monitors patterns over time (freshness, volume, distributions) to detect unexpected anomalies. You need both.

#Who is responsible for data quality?

Shared, but explicit: producers own the correctness of what they emit (ideally via contracts), data engineers own pipeline correctness, and analytics owners own metric definitions. Unclear ownership is a root cause of many quality issues.

#Do data contracts slow down application teams?

A little, at first. They add a check when changing data schemas. In return, they prevent downstream breakages that cost far more time to debug. Keep contracts lightweight and automate enforcement in CI.

#How many data tests are enough?

Start with keys and critical business rules on important models, then add tests whenever an incident reveals a gap. Coverage of what matters beats blanket tests on every column.