The stack, grouped by the decision it belongs to.
A list of logos proves nothing. What matters is knowing which third of the modern data stack you can safely skip. So these are grouped by the question each layer answers — with the trade-off that decides it, and a note on when the popular choice is the wrong one.
01 Move — ingestion & processing
How data gets in, and how it gets reshaped.
The first real decision on any platform: batch or stream. It's an operating-cost decision more than a technical one — streaming buys you latency and charges you in complexity, state management and on-call load.
When the data no longer fits on one machine
Spark remains the default for large batch work and is usually already in the building. Flink is the better answer when the workload is genuinely continuous and stateful rather than a schedule pretending to be a stream.
Wrong choice when: the dataset fits in memory on one large box. DuckDB or Polars will beat a Spark cluster on both runtime and your patience.
The buffer between systems that shouldn't know about each other
Kafka is the default log; Pulsar is worth considering for multi-tenant estates with many topics and tiered storage needs. The design work is topics, keys, partitions, retention and schema evolution — not the broker.
Wrong choice when: two services need to talk and you have one team. A queue you have to operate is a heavy price for decoupling you may not need.
02 Store — lakehouse, warehouse, cloud
Where the data lives, and who is allowed to read it cheaply.
The single most expensive decision on the page. Table format and partitioning determine your bill for years; the query engine is comparatively easy to change later — provided you picked an open format to begin with.
Transactions and schema evolution over files you own
Iceberg has become the safe default: broad engine support, good partition evolution, no vendor gravity. Delta is the natural fit inside Databricks; Hudi still wins for heavy upsert-and-incremental-read workloads.
Wrong choice when: you have no compaction strategy. An open table format with unmanaged small files is slower and dearer than the CSVs it replaced.
Making the data answerable by people who write SQL
Managed warehouses are worth their price when concurrency, governance and zero operations matter more than unit cost. Trino and DuckDB cover the cases where you want to query the lake directly without paying for a warehouse to hold it.
Wrong choice when: it's being used as a file system. Warehouse storage priced as compute is the most common line item nobody has looked at.
Sub-second queries over data that arrived seconds ago
A different job from a warehouse: high-cardinality filters, tight latency, live ingestion. ClickHouse for analytical breadth, Druid for time-series slice-and-dice, streaming databases where the query itself should be incremental.
Wrong choice when: the dashboard is refreshed by a human twice a day. That's a warehouse query with a cache in front of it.
The managed pieces that are worth not building
Managed services are used where they remove real operational burden and avoided where they quietly become the architecture. The rule of thumb: managed compute, yes; managed logic you can't test locally, carefully.
Wrong choice when: business logic ends up inside a drag-and-drop tool. It can't be reviewed, tested, or moved.
03 Model — orchestration, transformation, trust
Turning stored data into numbers people will act on.
This is where most of the trust in a platform is won or lost. Orchestration decides whether failures are recoverable; modelling decides whether two teams can agree on what a customer is.
What runs, in what order, and what happens when it fails
Dagster's asset model fits data platforms well and makes lineage a first-class concept. Airflow is everywhere and perfectly good when the team already runs it. Temporal is for long-running business workflows, not for nightly tables.
Where business logic is allowed to live
dbt is the default and the one your analysts likely already know. SQLMesh earns its place when column-level lineage, real change categorisation and cheap virtual environments would change how the team works.
Knowing something broke before a stakeholder does
Expectations that run inside the pipeline rather than beside it; lineage that answers "what does this feed" during an incident; and the same metrics and traces the rest of your infrastructure already emits.
04 Serve — AI, ML and applications
Getting the data to the thing that uses it.
A model, a dashboard, an API or a product feature. Different consumers, one requirement: the data arrives fresh, correct, and with the permissions still attached to it.
Retrieval, embeddings, features and evaluation
Vector storage chosen by corpus size and existing infrastructure — pgvector is the right answer more often than the market implies, because a database you already run beats a service you now have to operate.
Wrong choice when: there's no evaluation set. Without one you can't tell an improvement from a regression, and every change becomes an opinion.
The surface people and services actually touch
Services in front of the platform, internal tools for the team that operates it, and analytics interfaces built to stay fast at real data volumes rather than on a seeded demo.
Wrong choice when: a BI tool would do. Custom analytics UIs are a maintenance commitment, not a weekend.
05 Run — infrastructure & languages
Everything should be rebuildable from a repository.
If the platform can't be recreated from version control, it isn't a platform — it's a configuration someone remembers. Infrastructure as code is not a preference here, it's the deliverable.
Reproducible environments, boring deployments
Chosen for the layer, not for the résumé
Python and SQL for the overwhelming majority of data work. Scala or Java where a JVM engine makes it the honest choice. Rust or Go where a hot path genuinely needs it. TypeScript for anything with a browser attached.
06 Concepts, used carefully
Data mesh and data fabric, without the religion.
Both describe real problems. Data mesh is an organisational answer: domain teams own their data as a product, with contracts and SLAs, because a central team became the bottleneck. Data fabric is a technical answer: a metadata and governance layer over storage that's already scattered.
Both are worth borrowing from. Neither is worth adopting wholesale at a company with one data team — federated ownership without enough teams to federate is just a platform with extra ceremony. The useful parts, taken early: explicit data contracts, a real owner per dataset, and discoverability that doesn't depend on asking in Slack.
- Data contracts between producer and consumer teams
- A named owner for every dataset that matters
- Datasets treated as products, with an SLA attached
- A catalogue that answers "what feeds this" during an incident
- Self-serve access that doesn't route through one person
Next step
Which of these do you actually need?
Usually fewer than the vendor slide suggests. Thirty minutes on a call is enough to rule most of it out — and the ruling-out is the valuable part.