Taming Spark Event Logs: Cutting Azure Storage Costs by 90% Without Losing Observability

Maintaining observability in distributed data pipelines is non-negotiable. If a Spark job fails in production, you need the Spark History Server to replay the execution DAG, inspect task stragglers, and debug memory skew.

To power the History Server, Apache Spark writes a detailed Event Log to a persistent storage layer (like Azure ADLS Gen2) during job execution. However, out of the box, Spark’s default configurations for these event logs are designed for massive, long-running streaming clusters.

When applied to high-frequency, short-lived batch jobs (running on Kubernetes CronJobs every few hours), these defaults create a silent storage footprint explosion. In this post, I’ll walk through how we slashed our event log storage footprint by 90%, dropping our Azure RA-GRS storage costs significantly, all without sacrificing the fidelity of our debugging telemetry.

The Cost of Default Configurations

Our Hotpath consumer architecture relies on 2 primary Spark apps running 8 times a day via a 0 */3 * * * cron schedule. With Spark’s default settings, a single 3-hour micro-batch run was generating ~500 MB of uncompressed event logs.

Let’s do the math on that default setup:

• 500 MB per run × 16 runs per day = 8 GB per day.
• With a strict 90-day compliance retention policy, the steady-state storage sits at ~720 GB.
• On Azure ADLS Gen2 with RA-GRS (Read-Access Geo-Redundant Storage), 720 GB of highly transactional data (including geo-replication write transfer fees) starts to put unnecessary pressure on the cloud budget—specifically around $41/month just for logs.

The Trap of Rolling Event Logs

Our first instinct was to enable Spark 3.0’s rolling event log feature (spark.eventLog.rolling.enabled = true). In theory, this splits the monolithic event log into smaller, fixed-size segments.

However, in practice, this backfired spectacularly.

When we deployed rolling event logs to our production Kubernetes cluster on May 21st, we noticed an immediate and massive spike in Azure ADLS Gen2 transaction costs. Because our micro-batches were constantly churning through small tasks, the rolling log feature was aggressively committing tiny segments and metadata updates across the network.

According to our ADLS Gen2 Transactions by API name metrics, our AppendBlob and FlushFile operations jumped from a baseline of ~20,000 transactions per 6-hour window to over 43,600 transactions. The sheer volume of these network operations completely outpaced any storage capacity savings!

Furthermore, because our batch jobs were relatively short-lived (only a few minutes), they rarely hit the 128 MB rolling threshold gracefully. Enabling compaction to delete older segments also meant we would permanently lose task events that we needed for 90-day auditing.

We quickly realized that for high-frequency micro-batch architectures, rolling logs solve the wrong problem. Three days later, on May 24th, we reverted to single-file mode (spark.eventLog.rolling.enabled = false) and attacked the file size and transaction volume simultaneously.

The Fix: Buffering and ZSTD Compression

By default, Spark flushes event logs frequently. Every flush is an I/O operation to Azure, and the resulting JSON string blocks take up massive amounts of space. We implemented a two-pronged attack in our storage_handler.py to minimize both storage size and transaction volume:

(
    SparkSession.builder
    # ... other configs ...
    .config("spark.eventLog.enabled", "true")
    .config("spark.eventLog.dir", _event_log_dir)

    # 1. The 10 MB Buffer
    .config("spark.eventLog.buffer.kb", "10240")

    # 2. ZSTD Compression
    .config("spark.eventLog.compress", "true")
    .config("spark.eventLog.compression.codec", "zstd")

    # 3. Explicitly Disable Rolling
    .config("spark.eventLog.rolling.enabled", "false")
)

1. The JVM Heap Buffer (spark.eventLog.buffer.kb)

By setting the buffer to 10240 KB (10 MB), we force Spark to hold event log data in the JVM heap and batch it together before flushing it to ADLS Gen2. This dramatically reduces the number of small block appends across the network, saving on Azure transaction costs and smoothing out I/O spikes.

2. ZSTD Compression

Zstandard (ZSTD) is a modern compression algorithm that offers incredible compression ratios with minimal CPU overhead. By compressing the event logs in-flight, a 512 MB raw JSON log shrinks down to roughly 50 MB on disk.

The History Server Backend

Generating smaller logs is only half the battle. If your History Server isn’t configured correctly, downloading and parsing 90 days of logs will crash the container.

We deployed the History Server via Docker Compose with a highly optimized SPARK_HISTORY_OPTS configuration:

environment:
  - >
    SPARK_HISTORY_OPTS=
    -Dspark.history.fs.logDirectory=abfss://${SPARK_EVENT_LOG_CONTAINER}@...
    -Dspark.history.retainedApplications=1500
    -Dspark.history.ui.maxApplications=1500
    -Dspark.history.store.path=/tmp/spark-history-store
    -Dspark.history.store.maxDiskUsage=20g
    -Dspark.history.store.serializer=PROTOBUF
    -Dspark.history.fs.cleaner.enabled=true
    -Dspark.history.fs.cleaner.maxAge=90d

Key optimizations here include:

• PROTOBUF Serialization: By setting store.serializer=PROTOBUF, the internal LevelDB/RocksDB cache operates ~3x faster and smaller than the default JSON parsing (available in Spark 3.4+).
• Persistent Caching: We mounted /tmp/spark-history-store to a named Docker volume. If the container restarts, it doesn’t have to re-download and re-parse 72 GB of logs from Azure.
• Automated Janitor: The internal cleaner automatically deletes records older than 90 days, guaranteeing the UI never hits an OutOfMemoryError.

The Results

By combining the 10 MB JVM buffer with ZSTD compression, the steady-state 90-day storage footprint on Azure ADLS Gen2 plummeted from 720 GB to just 72 GB.

Metric	Before (Rolling Logs)	After (ZSTD + Buffer)	Impact
Log Size (per run)	~500 MB	~50 MB	-90%
90-Day Storage Vol	~720 GB	~72 GB	-90%
Storage Cost (RA-GRS)	~$41.04/mo	~$4.58/mo	-$36.46/mo
ADLS Transactions (per day)	~174,400	~75,200	-56%
Transaction API Cost	~$34.00/mo	~$14.66/mo	-$19.34/mo
TRUE TOTAL COST	~$75.04/mo	~$19.24/mo	-$55.80/mo

By explicitly disabling rolling logs and introducing the 10 MB JVM buffer with ZSTD compression, the steady-state 90-day storage footprint on Azure ADLS Gen2 plummeted from 720 GB to just 72 GB, and our runaway transaction APIs were cut in half.

The true total Azure cost (Storage Capacity + Write Transfer Bandwidth + Transaction APIs) dropped from $75/month to **$19/month** (a ~74% reduction). Over a year, this simple configuration tweak saves nearly $670 in cloud spend for just two consumer apps. When scaled across dozens of enterprise consumers, the savings compound massively.