Hi, I'm Amardeep
Lead Software Engineer · real-time data platforms
15+ years architecting large-scale distributed data platforms at Warner Media, Microsoft, and JPMorgan Chase. I build production streaming systems that process 10TB+/day at sub-100ms latency — and I write about Spark internals, streaming architectures, and the realities of running data platforms in the cloud.
What I Do
Data Platforms
Architecting large-scale batch and streaming platforms on Spark, Kafka, Databricks, and Delta Lake — from data contracts to production SLAs.
Real-Time Streaming
Building production streaming systems processing 10TB+/day with sub-100ms p99 latency, exactly-once semantics, and auto-scaling.
Performance & Cost
Deep Spark internals tuning and platform optimization — data skew, AQE, broadcast joins — driving major cost and latency wins.