Professional Summary
Staff Data Engineer with 15+ years architecting and leading large-scale distributed data platforms at Warner Media, Microsoft, and JPMorgan Chase. Expert in building production streaming systems processing 10TB+/day at sub-100ms latency using Spark, Kafka, and cloud-native architectures. Track record of driving strategic cost optimization ($700K+/year saved), leading cross-functional migrations affecting 15+ teams, and building reusable frameworks adopted across organizations. Deep expertise in Spark internals, real-time data architectures, and cloud data platform evolution (AWS, Databricks, Azure).
Key Achievements
- Led cross-functional data platform migrations impacting 15+ teams and 80+ production pipelines with zero downtime
- Built and scaled real-time streaming systems processing 10TB+/day with <100ms p99 latency serving 50M+ users
- Drove $700K+ in annual cost savings through platform optimization initiatives
- Architected reusable frameworks reducing time-to-production by 85% for streaming use cases
- Established technical authority across 40+ engineers in Spark, Kafka, and cloud data platforms
Experience
Senior Data Engineer
Warner Media – Discovery London, UK · Feb 2022 – Present- Led enterprise migration from Snowflake to Databricks — framework and governance model adopted by 4 teams (15+ engineers, 80+ pipelines); 50% cost reduction ($500K/yr) and 3x query performance.
- Architected the HBO Max AdTech data platform serving 50M+ daily users across 60+ countries; Lambda architecture (Databricks + Kafka/Spark Structured Streaming) processing 5TB/day at 99.95% SLA, cutting data latency from 4 hours to 15 minutes and enabling $5M+ in targeted ad revenue.
- Designed the RTCM real-time streaming framework — 500K events/sec at <100ms p99, exactly-once, auto-scaling; adopted by 3 teams (12+ pipelines), cutting time-to-production from 6 weeks to 1.
- Drove a Spark performance program (data-skew salting, broadcast joins, AQE): 30% cluster cost cut ($200K/yr), 40% faster jobs, 95% fewer OOM errors; trained 20+ engineers.
Senior Software Engineer
Microsoft Bangalore, India · Jun 2021 – Dec 2021- Core contributor to the Spark execution engine powering Azure Data Factory (4000+ enterprise clients, 100+ PB/month).
- Built a custom Spark Listener framework streaming 1M+ metrics/min to the ADF UI at <5s latency, cutting support tickets by 35%.
- Designed a high-cardinality metrics aggregation pipeline (HyperLogLog, T-Digest) holding sub-second queries at 10x peak load.
Senior Data Engineer
JPMorgan Chase & Co Bangalore, India · Mar 2020 – Apr 2021- Led a near-real-time risk analytics platform processing 2M+ trades/day from 15+ sources using Spark Structured Streaming + Kafka with event-time aggregations, stream-stream joins, and exactly-once multi-sink writes.
- Cut risk-calculation latency from 30 minutes to under 2 — 15x faster decisions — supporting $500B+ in daily trading volume.
- Migrated legacy DStream pipelines to Structured Streaming; an Avro schema registry gave 40% compression and safe schema evolution. Playbook adopted by 3 teams.
Technical Lead
Emids Technologies Bangalore, India · May 2016 – Jun 2019- Architected the Enterprise Data Hub — a multi-tenant Cassandra healthcare data lake serving 8 insurers, 100M+ patient records, and 500GB/day ingestion; led a team of 6.
- Built the processing framework: schema validation with PII masking, dedup/cleanse, CDC, and graph-based (GraphX Connected Components) patient identity resolution at 98% accuracy — cutting an 8-hour job to 45 minutes.
- Designed a Java/Spring Boot microservices layer with Redis + Cassandra caching at <50ms p95 for 10K+ req/sec.
Senior Software Engineer
Amadeus Labs Bangalore, India · Sep 2013 – Nov 2015- Built a Spark/Flume pipeline for central logging — ingesting and parsing semi-structured logs and traces with automated anomaly detection.
- Deployed a scalable Flume-based collection system converting logs to Parquet with schema enforcement for downstream analytics.
Technical Skills
Core Strengths:
Distributed Systems Architecture, Real-Time Streaming, Data Platform Modernization, Performance Optimization
Languages:
Java, Scala, Python, Golang
Data Engineering:
Apache Spark (Batch & Structured Streaming), Kafka, Flink, Delta Lake, Snowflake, Databricks, Airflow
Databases:
Cassandra, HBase, PostgreSQL, MySQL, Redis
Cloud Platforms:
AWS (S3, EMR, Kinesis, Redshift, Glue), Databricks (Unity Catalog, Delta Live Tables), Azure (ADF, Synapse)
DevOps & Tools:
Kubernetes, Docker, Terraform, Jenkins, GitLab CI/CD, Elasticsearch
Frameworks & Platforms
RTCM Streaming Framework
Spark Structured Streaming, Kafka
Reusable event-processing platform running 500K events/sec at <100ms p99 with exactly-once semantics and auto-scaling. Adopted by 3 teams, reducing time-to-production by 85%.
Databricks Migration Framework
Databricks, Delta Lake, Unity Catalog
Migration patterns and governance model used by 4 teams to move 80+ pipelines off Snowflake with zero downtime and a 50% cost reduction.
Spark Performance Toolkit
Apache Spark, AQE
Best practices and automated tuning scripts deployed across 40+ engineers — codifying data-skew handling, broadcast joins, and AQE tuning into repeatable runbooks.