The Silent Timezone Shift: Debugging PySpark's .collect() Offset Bug
How querying a UTC timestamp via PySpark's .collect() magically gained a +7 hour offset, and how to safely serialize dates from the JVM to Python.
Writings
Engineering thoughts, guides, and architectural decisions.
Beyond DISTINCT: The Smart Deduplication Pattern Using Window Functions
How standard DISTINCT logic fails on edge cases, and why ROW_NUMBER() window functions are the ultimate tool for intelligent data deduplication.
The Spark 1582 Anomaly: Handling Legacy Dates in Parquet
How a dummy date of 0001-01-01 crashed our production pipeline, and why Spark 3.x enforces the Proleptic Gregorian calendar.
Taming Spark Event Logs: Cutting Azure Storage Costs by 90% Without Losing Observability
A deep dive into Spark History Server configuration, and how we saved on Azure ADLS Gen2 costs by leveraging ZSTD compression and JVM heap buffering.
Delta Lake Liquid Clustering Benchmark
An empirical deep-dive into the impact of Liquid Clustering on upsert (MERGE) performance using a real-world telematics dataset.
Taming the Data Swamp: The Impact of OPTIMIZE, ZORDER, and VACUUM on Delta Tables
An empirical deep-dive into resolving Write Amplification and Small File Problems in Delta Lake on a managed cloud Spark cluster with local M2 benchmarks.