Data
Engineering
CAT EG O RIES
01
Programming
&
Query
Languages
02
Databases
&
Operational
Storage
03
Warehouses
&
Lakehouse
Platforms
04
Data
Modeling
&
Architecture
05
ETL
,
ELT
&
Transformation
06
Orchestration
&
Workflow
07
Big
Data
&
Distributed
Processing
08
Streaming
&
Event
-
Driven
Data
09
Cloud
Platforms
&
Services
10
File
Formats
&
Serialization
11
Data
Quality
,
Testing
&
Observability
12
Governance
,
Security
&
Privacy
13
DevOps
,
Platform
&
Infrastructure
14
Analytics
,
BI
&
ML
Enablement
15
Core
Engineering
Fundamentals
01
Programming
&
Query
Languages
The
languages
you
write
pipelines
,
transformations
,
and
ad
-
hoc
analysis
in
.
Python
SQL
Scala
Java
Go
Bash
/
Shell
scripting
R
Rust
Kotlin
PySpark
T
-
SQL
PL
/
SQL
BigQuery
Standard
SQL
Cypher
02
Databases
&
Operational
Storage
Systems
that
hold
source
data
and
serve
live
application
workloads
.
PostgreSQL
MySQL
/
MariaDB
Microsoft
SQL
Server
Oracle
Database
MongoDB
Apache
Cassandra
Redis
Amazon
DynamoDB
Elasticsearch
/
OpenSearch
Neo
4
j
ClickHouse
Couchbase
Firestore
Amazon
Aurora
03
Warehouses
&
Lakehouse
Platforms
Analytical
engines
where
modeled
data
lands
and
is
queried
at
scale
.
Snowflake
Google
BigQuery
Amazon
Redshift
Databricks
SQL
Warehouse
Microsoft
Fabric
Azure
Synapse
Analytics
Teradata
DuckDB
Apache
Iceberg
Delta
Lake
Apache
Hudi
Trino
Vertica
1 / 4
04
Data
Modeling
&
Architecture
How
data
is
shaped
,
layered
,
and
organized
so
it
stays
trustworthy
at
scale
.
Dimensional
modeling
(
Kimball
)
Third
normal
form
(
Inmon
)
Data
Vault
2.0
Star
schema
Snowflake
schema
Slowly
changing
dimensions
(
SCD
1–6)
Activity
Schema
Normalization
&
denormalization
Medallion
architecture
Data
mesh
Lambda
architecture
Kappa
architecture
Lakehouse
architecture
One
Big
Table
design
Grain
definition
Surrogate
&
natural
keys
05
ETL
,
ELT
&
Transformation
Moving
data
between
systems
and
expressing
business
logic
as
version
-
controlled
code
.
dbt
Core
dbt
Cloud
Jinja
&
dbt
macros
SQLMesh
Apache
Airflow
Dagster
Prefect
Fivetran
Airbyte
Meltano
Matillion
Talend
Informatica
PowerCenter
SSIS
Apache
NiFi
Stored
procedures
06
Orchestration
&
Workflow
Scheduling
,
dependency
management
,
retries
,
and
backfills
across
the
platform
.
Airflow
DAG
design
Dagster
software
-
defined
assets
Prefect
flows
Argo
Workflows
Temporal
Apache
Luigi
Control
-
M
Azure
Data
Factory
pipelines
AWS
Step
Functions
Kubernetes
CronJobs
Cron
&
systemd
timers
Event
-
triggered
scheduling
Dependency
&
backfill
management
07
Big
Data
&
Distributed
Processing
Frameworks
for
processing
volumes
that
do
not
fit
on
a
single
machine
.
Apache
Spark
PySpark
Spark
SQL
Spark
Structured
Streaming
Hadoop
HDFS
MapReduce
Apache
Hive
Presto
/
Trino
Apache
Flink
Apache
Beam
Dask
Ray
Apache
Sqoop
Apache
Pig
YARN
2 / 4
08
Streaming
&
Event
-
Driven
Data
Continuous
pipelines
,
change
data
capture
,
and
real
-
time
delivery
.
Apache
Kafka
Confluent
Platform
Redpanda
Amazon
Kinesis
Google
Cloud
Pub
/
Sub
Azure
Event
Hubs
Apache
Pulsar
RabbitMQ
Debezium
CDC
Kafka
Connect
Kafka
Streams
Schema
Registry
Avro
&
Protobuf
schemas
Windowed
aggregation
&
watermarking
Delivery
semantics
(
at
-
least
-
once
/
exactly
-
once
)
09
Cloud
Platforms
&
Services
The
managed
infrastructure
you
will
actually
deploy
pipelines
on
.
AWS
S
3
AWS
Glue
AWS
EMR
AWS
Lambda
AWS
Lake
Formation
AWS
Athena
AWS
Kinesis
Data
Firehose
GCP
BigQuery
GCP
Dataflow
GCP
Dataproc
GCP
Cloud
Composer
GCP
Dataplex
Azure
Data
Factory
Azure
Data
Lake
Storage
Gen
2
Azure
Databricks
Azure
Functions
10
File
Formats
&
Serialization
How
data
is
written
to
disk
and
moved
efficiently
between
systems
.
Apache
Parquet
Apache
ORC
Avro
Delta
Lake
tables
Apache
Iceberg
tables
JSON
JSONL
/
NDJSON
CSV
XML
Apache
Arrow
Protocol
Buffers
Apache
Thrift
11
Data
Quality
,
Testing
&
Observability
Catching
bad
data
before
stakeholders
do
—
and
knowing
why
it
broke
.
Great
Expectations
dbt
tests
Soda
Elementary
Monte
Carlo
Deequ
Anomalo
Data
contracts
Schema
evolution
&
drift
detection
Data
profiling
Reconciliation
&
row
-
count
checks
Freshness
&
SLA
monitoring
Column
-
level
lineage
Root
-
cause
analysis
Incident
runbooks
3 / 4
12
Governance
,
Security
&
Privacy
Catalogs
,
access
control
,
compliance
,
and
the
metadata
that
holds
it
together
.
Data
catalogs
(
DataHub
,
Amundsen
,
Atlan
)
Metadata
management
Business
glossary
PII
detection
&
classification
GDPR
&
CCPA
compliance
RBAC
&
ABAC
Column
-
level
encryption
Tokenization
&
data
masking
Row
-
level
security
policies
Audit
logging
Retention
&
right
-
to
-
delete
workflows
Master
data
management
Data
stewardship
&
ownership
Consent
management
13
DevOps
,
Platform
&
Infrastructure
Shipping
,
running
,
and
monitoring
the
data
platform
itself
.
Git
&
GitHub
Docker
Kubernetes
Terraform
Pulumi
AWS
CloudFormation
Helm
Ansible
GitHub
Actions
Jenkins
GitLab
CI
Linux
administration
Prometheus
&
Grafana
Secrets
management
(
Vault
)
FinOps
&
cost
optimization
On
-
call
&
incident
response
14
Analytics
,
BI
&
ML
Enablement
Serving
clean
data
to
analysts
,
dashboards
,
and
machine
learning
models
.
Tableau
Power
BI
Looker
Metabase
Apache
Superset
Semantic
layer
(
dbt
Semantic
Layer
,
Cube
)
Feature
stores
(
Feast
,
Tecton
)
MLflow
Vector
databases
(
pgvector
,
Pinecone
)
RAG
pipelines
Reverse
ETL
(
Census
,
Hightouch
)
Embedded
analytics
Metric
definitions
&
governance
Experimentation
pipelines
Real
-
time
dashboards
15
Core
Engineering
Fundamentals
The
concepts
that
outlast
every
tool
on
this
list
.
Distributed
systems
CAP
theorem
&
consistency
models
Partitioning
&
clustering
Indexing
strategies
Query
optimization
&
execution
plans
Concurrency
&
parallelism
Idempotency
Exactly
-
once
processing
Data
structures
&
algorithms
Batch
vs
.
micro
-
batch
vs
.
streaming
Capacity
planning
Cost
-
per
-
query
tuning
Networking
&
IAM
basics
Retry
&
error
-
handling
design
Documentation
&
runbooks
Agile
delivery
4 / 4
