About Me

Staff Data Engineer turning data chaos into platforms teams can build on.

Hello, I'm Amardeep Chaubey

I'm a Staff Data Engineer with over 15 years building and leading large-scale distributed data platforms. My work centers on production streaming systems — the kind that process 10TB+ a day at sub-100ms latency — across Warner Media's HBO Max AdTech, Microsoft's Azure Data Factory, and JPMorgan Chase's trading risk systems.

I care most about the unglamorous parts: Spark internals, data skew, exactly-once semantics, cost optimization, and the migration frameworks that let a dozen teams move without downtime. Along the way I've driven $700K+ in annual savings and cut time-to-production by 85% with reusable streaming frameworks.

I also enjoy teaching — mentoring engineers into senior and lead roles, running workshops on Spark and streaming, and now writing this blog to share what I've learned the hard way. Based in London.

Amardeep Chaubey

Skills & Technologies

Languages

  • Java
  • Scala
  • Python
  • Golang

Data Engineering

  • Apache Spark (Batch & Structured Streaming)
  • Kafka / Flink
  • Delta Lake / Databricks
  • Snowflake
  • Airflow

Cloud Platforms

  • AWS (S3, EMR, Kinesis, Redshift, Glue)
  • Databricks (Unity Catalog, DLT)
  • Azure (ADF, Synapse)

Databases

  • Cassandra
  • HBase
  • PostgreSQL / MySQL
  • Redis

DevOps & Tools

  • Kubernetes / Docker
  • Terraform
  • Jenkins / GitLab CI/CD
  • Elasticsearch

What Drives Me

Reliability

Zero-downtime migrations and 99.95% SLAs aren't luck — they come from boring, deliberate engineering.

Performance

A 15x latency win or a 50% cost cut usually hides in the internals. I like digging there.

Mentorship

The best systems outlast me because the team around them got stronger. Growing engineers is the work.

Simplicity

Unified batch/streaming APIs, fewer moving parts, clearer data contracts — complexity is a cost, not a badge.