Platform · 2014–now · Cloud Scale
Apache Spark
In-memory data processing that made Hadoop feel overnight. Still the batch/stream workhorse under many lakehouses.
Spark stuck by meeting analysts where Python and SQL already lived while beating MapReduce ergonomics. It is not magic — shuffle still hurts — but it ships pipelines. The fad was calling every CSV a lake; Spark survived the hangover.
Context
Split everything, then pay for the glue
Hyperscale patterns escaped the companies that needed them. Containers unified packaging; Kubernetes became the cloud OS; microservices and NoSQL were sold as defaults. Mobile-first stuck because screens changed. Cargo-cult distributed systems stuck around as invoices. The durable move was packaging and ops maturity — not rewriting every app into a mesh.
Compare with
Related
Platform · 2006–2018
Hadoop Everywhere
MapReduce as lifestyle. The elephant in rooms that needed a spreadsheet.
$ Enterprises stood up Hadoop clusters for gigabytes of data that fit on one Postgres instance. Hadoop admins, ZooKeeper nightmares, and ETL rewrites burned millions before Spark and cloud warehouses retired the elephant.
Language · 2004–now
Scala
JVM language with functional teeth. Spark and Kafka culture kept it relevant long after the Twitter hype cooled.
Language · 1991–now
Python
From scripting glue to the default language of data and AI. Readability as a growth strategy.
Platform · 2011–now
Apache Kafka
Durable log as nervous system. Event-driven architecture with real operational teeth.