Taming Spark Data Skew with Salting
If you’ve ever watched a Spark job sit at 199/200 tasks complete for twenty minutes, you’ve met data skew. One partition got most of the rows, and one executor is doing all the work while the rest ...
Read MoreWhy Your p99 Latency Lies to You
Someone asks how fast your pipeline is. You say “p99 is 80ms” and everyone nods. But that one number is quietly lying to you — or at least letting you lie to yourself.
Read More