Big
Data
Technical
Skills
CO RE
SKI LLS
AT
A
G LAN CE
Python
SQL
&
query
tuning
Scala
Apache
Spark
Apache
Kafka
Apache
Flink
Hadoop
&
HDFS
Snowflake
Google
BigQuery
Databricks
Delta
Lake
&
Iceberg
Apache
Airflow
dbt
Dimensional
modeling
Parquet
·
ORC
·
Avro
Trino
/
Presto
ClickHouse
Kubernetes
&
Docker
Terraform
AWS
EMR
·
Glue
·
Athena
Great
Expectations
MLflow
Tableau
&
Power
BI
Spark
performance
tuning
SKI LLS
BY
CAT E G O RY
01
Programming
&
Scripting
Languages
7
SKILLS
Python
pandas
,
NumPy
,
PySpark
,
async
I
/
O
,
packaging
SQL
ANSI
SQL
,
window
functions
,
CTEs
,
recursive
queries
,
tuning
Scala
Functional
programming
for
Spark
and
Flink
jobs
Java
JVM
services
,
Hadoop
ecosystem
libraries
,
Kafka
clients
R
Statistical
modelling
,
tidyverse
,
R
Markdown
reporting
Bash
/
Shell
Job
automation
,
log
parsing
,
cron
orchestration
Go
High
-
throughput
ingestion
services
and
CLI
tooling
1 / 7
02
Distributed
Processing
Frameworks
9
SKILLS
Apache
Spark
RDDs
,
DataFrames
,
Spark
SQL
,
Structured
Streaming
Hadoop
MapReduce
Batch
processing
over
HDFS
with
custom
mappers
and
reducers
Apache
Flink
Stateful
stream
processing
with
exactly
-
once
semantics
Apache
Beam
Unified
batch
and
streaming
pipelines
with
portable
runners
Apache
Storm
Low
-
latency
event
processing
topologies
Kafka
Streams
In
-
application
stream
processing
with
state
stores
Apache
Hive
SQL
-
on
-
Hadoop
with
Tez
and
LLAP
execution
Trino
/
Presto
Federated
SQL
across
heterogeneous
data
sources
Dask
&
Ray
Parallel
Python
and
distributed
compute
graphs
03
Data
Storage
&
File
Formats
7
SKILLS
HDFS
Distributed
block
storage
,
replication
,
NameNode
architecture
Object
Storage
Amazon
S
3,
Azure
Data
Lake
Storage
Gen
2,
Google
Cloud
Storage
Columnar
Formats
Parquet
and
ORC
with
predicate
pushdown
and
encoding
Row
Formats
Avro
,
JSON
,
and
Protobuf
for
schema
evolution
Lakehouse
Tables
Delta
Lake
,
Apache
Iceberg
,
Apache
Hudi
Key
-
Value
Stores
HBase
,
Cassandra
,
RocksDB
state
backends
Compression
Snappy
,
Zstandard
,
Gzip
,
LZO
trade
-
offs
for
scan
speed
2 / 7
04
Data
Ingestion
&
Streaming
7
SKILLS
Apache
Kafka
Topics
,
partitions
,
consumer
groups
,
offsets
,
exactly
-
once
delivery
Amazon
Kinesis
Data
Streams
,
Firehose
delivery
,
Managed
Flink
Google
Pub
/
Sub
At
-
least
-
once
messaging
at
scale
with
ordering
keys
Azure
Event
Hubs
Kafka
-
compatible
event
ingestion
and
capture
Apache
NiFi
Flow
-
based
data
routing
,
transformation
,
and
provenance
Change
Data
Capture
Debezium
,
Oracle
GoldenGate
,
AWS
DMS
,
log
-
based
replication
Batch
Connectors
Airbyte
,
Fivetran
,
Sqoop
,
Flume
05
Databases
&
Warehouses
9
SKILLS
Snowflake
Virtual
warehouses
,
time
travel
,
Snowpark
,
cloning
Google
BigQuery
Serverless
SQL
,
slots
,
partitioning
,
BI
Engine
Amazon
Redshift
RA
3
clusters
,
Spectrum
,
concurrency
scaling
,
sort
keys
Databricks
SQL
Lakehouse
warehouse
over
Delta
tables
and
Unity
Catalog
ClickHouse
Real
-
time
OLAP
at
sub
-
second
query
latency
Druid
&
Pinot
User
-
facing
analytics
on
high
-
volume
event
streams
Relational
Engines
PostgreSQL
,
MySQL
,
Oracle
,
SQL
Server
NoSQL
MongoDB
,
DynamoDB
,
Cassandra
,
Redis
Search
Engines
Elasticsearch
,
OpenSearch
,
Apache
Solr
3 / 7
06
Data
Modeling
&
Transformation
7
SKILLS
Dimensional
Modeling
Star
and
snowflake
schemas
,
fact
and
dimension
design
Slowly
Changing
Dimensions
Type
1, 2, 3
and
6
patterns
with
history
tracking
Data
Vault
2.0
Hubs
,
links
,
satellites
,
and
raw
vault
staging
Normalization
3
NF
design
for
operational
stores
and
reference
data
dbt
Models
,
macros
,
tests
,
incremental
and
snapshot
strategies
ELT
vs
ETL
Pushdown
transformation
inside
the
warehouse
engine
Schema
Evolution
Backward
and
forward
compatible
data
contracts
07
Orchestration
&
Workflow
7
SKILLS
Apache
Airflow
DAGs
,
operators
,
sensors
,
backfills
,
task
dependencies
Dagster
Software
-
defined
assets
and
typed
I
/
O
boundaries
Prefect
Dynamic
,
Python
-
native
flows
with
retries
and
caching
Argo
Workflows
Kubernetes
-
native
pipeline
execution
and
artifacts
Managed
Orchestration
Azure
Data
Factory
,
AWS
Glue
Workflows
,
Cloud
Composer
Legacy
Schedulers
Luigi
,
Oozie
,
Control
-
M
job
dependency
management
Scheduling
Strategy
Cron
,
backfills
,
SLA
alerting
,
idempotent
reruns
08
Cloud
Platforms
&
Managed
Services
7
SKILLS
AWS
EMR
,
Glue
,
Athena
,
Redshift
,
Lake
Formation
,
MSK
Microsoft
Azure
Synapse
,
Databricks
,
Data
Factory
,
Event
Hubs
,
Fabric
Google
Cloud
BigQuery
,
Dataflow
,
Dataproc
,
Pub
/
Sub
,
Dataplex
Databricks
Platform
Unity
Catalog
,
Delta
Live
Tables
,
Workflows
,
clusters
Containers
Docker
,
Kubernetes
,
Helm
for
portable
job
packaging
Infrastructure
as
Code
Terraform
,
CloudFormation
,
Pulumi
Cost
Management
Spot
fleets
,
auto
-
scaling
,
storage
tiering
,
chargeback
4 / 7
09
Query
Performance
&
Optimization
7
SKILLS
Partitioning
&
Bucketing
Scan
pruning
and
co
-
located
joins
on
large
tables
Join
Strategies
Broadcast
,
sort
-
merge
,
and
shuffle
hash
join
selection
Skew
Handling
Salting
,
adaptive
skew
join
,
repartitioning
hot
keys
Shuffle
Tuning
Memory
fractions
,
serializers
,
spill
reduction
File
Layout
Small
-
file
compaction
,
Z
-
ordering
,
clustering
keys
Query
Plans
EXPLAIN
plans
,
predicate
pushdown
,
table
statistics
Vectorization
&
Caching
Columnar
execution
engines
and
result
caching
10
Data
Quality
,
Governance
&
Lineage
7
SKILLS
Data
Quality
Testing
Great
Expectations
,
Soda
,
dbt
tests
,
contract
enforcement
Data
Observability
Monte
Carlo
,
Elementary
,
freshness
and
volume
anomaly
detection
Data
Catalogs
DataHub
,
Amundsen
,
Unity
Catalog
,
Glue
Data
Catalog
Column
-
Level
Lineage
Impact
analysis
and
root
-
cause
tracing
across
pipelines
Master
Data
Management
Golden
records
,
matching
,
and
entity
resolution
Privacy
Compliance
GDPR
,
CCPA
,
HIPAA
,
PII
masking
,
tokenization
Access
Control
Row
-
and
column
-
level
security
,
RBAC
and
ABAC
policies
11
Machine
Learning
&
Advanced
Analytics
7
SKILLS
ML
Libraries
scikit
-
learn
,
XGBoost
,
LightGBM
,
Spark
MLlib
Deep
Learning
TensorFlow
and
PyTorch
with
distributed
training
Feature
Stores
Feast
,
Tecton
,
Databricks
Feature
Store
,
online
/
offline
parity
MLOps
MLflow
,
Kubeflow
,
model
registries
,
drift
monitoring
Vector
Search
Embeddings
,
FAISS
,
Pinecone
,
pgvector
Experimentation
A
/
B
testing
,
causal
inference
,
uplift
modelling
Notebook
Workflows
Jupyter
,
Databricks
notebooks
,
papermill
,
reproducibility
5 / 7
12
Real
-
Time
&
Streaming
Analytics
6
SKILLS
Event
-
Time
Processing
Watermarks
,
windowing
,
late
-
data
handling
Exactly
-
Once
Semantics
Idempotent
sinks
and
transactional
writes
Stream
-
Batch
Unification
Lambda
vs
Kappa
architecture
trade
-
offs
Change
Streams
CDC
into
the
lakehouse
with
merge
and
upsert
patterns
Low
-
Latency
Serving
Druid
,
Pinot
,
ClickHouse
,
Redis
for
sub
-
second
reads
Alerting
Pipelines
Thresholds
,
complex
event
processing
,
notification
routing
13
Security
&
Reliability
6
SKILLS
Authentication
Kerberos
,
OAuth
2.0,
SSO
/
SAML
,
IAM
roles
and
federation
Encryption
TLS
in
transit
,
KMS
and
AES
-256
at
rest
Network
Isolation
VPC
design
,
private
endpoints
,
security
groups
Secrets
Management
HashiCorp
Vault
,
AWS
Secrets
Manager
,
key
rotation
High
Availability
Replication
,
multi
-
AZ
deployment
,
disaster
recovery
drills
Monitoring
Prometheus
,
Grafana
,
Datadog
,
CloudWatch
,
on
-
call
runbooks
14
Business
Intelligence
&
Visualization
6
SKILLS
Tableau
Extracts
,
calculated
fields
,
dashboard
design
Power
BI
DAX
,
dataflows
,
composite
models
,
row
-
level
security
Looker
&
Looker
Studio
LookML
models
and
governed
metric
definitions
Open
-
Source
BI
Apache
Superset
,
Metabase
,
Redash
Grafana
Operational
time
-
series
dashboards
and
alert
rules
Visualization
Craft
Chart
selection
,
color
accessibility
,
data
storytelling
6 / 7
15
Professional
&
Delivery
Skills
6
SKILLS
Data
Architecture
Lakehouse
,
data
mesh
,
and
data
fabric
trade
-
offs
Requirements
Gathering
Stakeholder
interviews
,
KPI
definition
,
scoping
Agile
Delivery
Scrum
,
Kanban
,
sprint
estimation
,
release
planning
Documentation
Data
dictionaries
,
runbooks
,
architecture
decision
records
Mentoring
&
Code
Review
Engineering
standards
,
pairing
,
onboarding
Stakeholder
Communication
Translating
pipeline
work
into
business
outcomes
7 / 7
