Pipelines, warehouses, streaming, data modeling and the infrastructure that turns raw data into something useful.
What data engineers do and how the modern data stack works — ingestion, storage, transformation, orchestration, quality and serving — plus the skills and tools involved.
ETL vs ELT explained — how extract-transform-load and extract-load-transform pipelines differ, their pros and cons, costs, governance implications and how to choose.
Compare data warehouses, data lakes and lakehouses — architecture, open table formats, cost, performance, governance and which one fits your analytics and AI workloads.
How Apache Kafka works — the distributed log, topics, partitions, offsets, consumer groups, replication, delivery guarantees, and real-world use cases and pitfalls.
Batch vs stream processing explained — latency, complexity, cost, windowing, late data, exactly-once semantics, Lambda and Kappa architectures, and how to decide.
Learn dimensional data modeling — star vs snowflake schemas, fact and dimension tables, grain, slowly changing dimensions, and how to model data for BI and analytics.
How to build reliable data pipelines with Apache Airflow — DAGs, tasks, scheduling, idempotency, backfills, retries, sensors, testing and common anti-patterns.
How to build trustworthy data — the dimensions of data quality, pipeline tests, data contracts between producers and consumers, anomaly detection, incident response and ownership.
What change data capture is, log-based vs query-based CDC, how tools like Debezium read database logs, use cases, the outbox pattern, and pitfalls like schema changes and deletes.
What dbt is and how it works — SQL models, ref(), materializations, incremental models, tests, documentation, lineage and best practices for structuring a dbt project.