The Spark 1582 Anomaly: Handling Legacy Dates in Parquet

Data pipelines are only as resilient as the weirdest piece of data they ingest. Recently, our production truckscale consumer pipeline crashed out of nowhere, bringing data ingestion to a grinding halt.

The culprit? A single dummy date: 0001-01-01.

When we inspected the Kubernetes logs, we were greeted with a massive stack trace leading up to a SparkUpgradeException. The error explicitly complained about writing dates before the year 1582.

If you are a data engineer moving data from legacy SQL Server databases into Parquet format using Apache Spark 3.x, you will eventually encounter this bizarre anomaly. Here is a reproducible example of why it happens and how to fix it.

Reproducing the Crash

To demonstrate this, we built a standalone PySpark script that attempts to write the date 0001-01-01:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("1582-Anomaly").getOrCreate()

# Create dummy data with year 0001
data = [("truck_1", "0001-01-01 00:00:00")]
df = spark.createDataFrame(data, ["truck_id", "timestamp_str"])

# Cast string to actual timestamp
df = df.withColumn("event_time", df["timestamp_str"].cast("timestamp"))

# Attempting to write this to Parquet will violently crash Spark 3.x
df.write.format("parquet").save("/tmp/anomaly_test")

When you execute this code in Spark 3.x, it throws this massive exception:

[CAUGHT EXCEPTION]
An error occurred while calling o48.save.
: org.apache.spark.SparkException: Job aborted due to stage failure...
Caused by: org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows...
Caused by: org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.WRITE_ANCIENT_DATETIME] You may get a different result due to the upgrading to Spark >= 3.0:
writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z
into Parquet INT96 files can be dangerous...

The Proleptic Gregorian Calendar Shift

To understand this error, you need a brief history lesson.

In October 1582, the world transitioned from the Julian calendar to the Gregorian calendar to correct a drift in the solar year. When this transition occurred, 10 days were essentially skipped (October 4th was followed immediately by October 15th).

In Spark 2.x and earlier versions, dates and timestamps were calculated using a hybrid calendar (Julian before 1582, Gregorian after). However, starting in Spark 3.0, the community made a massive shift to the Proleptic Gregorian calendar. This calendar extends the Gregorian rules backward infinitely, pretending the Gregorian calendar was always in use.

Why does this matter? Because if your source system (like a legacy ERP or an old SQL Server database) sends a dummy date of 0001-01-01 (a common default value), Spark 2.x and Spark 3.x will literally interpret that date as two different points in time due to the missing 10 days in 1582.

When you try to write this data to Parquet in Spark 3.x, Spark detects this ambiguity and violently throws a SparkUpgradeException to prevent silent data corruption.

The Fix: Rebase Modes

Spark gives you three options to handle this via configuration: EXCEPTION (the default), LEGACY, and CORRECTED.

Since our goal is forward compatibility and we want to safely write this data without crashing the pipeline, the solution is to explicitly tell Spark to rebase these ancient dates using the CORRECTED mode.

In our Python consumer, we injected these configurations directly into the SparkSession builder:

(
    SparkSession.builder
    .appName("truckscale-consumer")

    # Handle dates/timestamps before 1582
    .config("spark.sql.parquet.int96RebaseModeInWrite", "CORRECTED")
    .config("spark.sql.parquet.int96RebaseModeInRead", "CORRECTED")
    .config("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")
    .config("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")

    .getOrCreate()
)

What does CORRECTED actually do?

By setting the rebase mode to CORRECTED, Spark will read and write the ancient dates using the new Proleptic Gregorian calendar rules. It acknowledges the shift and adjusts the underlying Julian day calculation so that 0001-01-01 remains 0001-01-01 when read back in Spark 3.x, completely bypassing the exception.

--- Applying the Fix ---
Attempting to write ancient date to Parquet WITH Rebase Mode = CORRECTED...
[SUCCESS] Parquet file written successfully!

(Note: You only need int96RebaseMode if you are dealing with legacy INT96 timestamps from systems like Impala or old Hive tables. We include them as a standard boilerplate just to be safe).

WARNING

The Second Trap: Date Parsing
We fixed the Parquet writer, but the Proleptic Gregorian calendar shift also broke string parsing in Spark 3.x! If you use functions like to_date() or unix_timestamp() on weird legacy date strings (e.g., 0001-01-01), Spark’s strict Java 8 time parser will throw a similar SparkUpgradeException. To fix the parser side of this anomaly, configure spark.conf.set("spark.sql.legacy.timeParserPolicy", "LEGACY").

Conclusion

If your pipelines ingest data from external systems, never trust the dates. Source systems love using 0001-01-01 or 1000-01-01 as placeholder defaults for NULLs.

By explicitly setting your Parquet rebase modes to CORRECTED in your Spark session, you armor your pipelines against the 1582 calendar anomaly and ensure your ingestion never crashes over a 2,000-year-old dummy value.