diff --git a/docs/sql-data-sources-avro.md b/docs/sql-data-sources-avro.md index eeb4bffa57d1..6e23867ad38f 100644 --- a/docs/sql-data-sources-avro.md +++ b/docs/sql-data-sources-avro.md @@ -283,6 +283,19 @@ Data source options of Avro can be set via: function from_avro + + datetimeRebaseMode + The SQL config spark.sql.avro .datetimeRebaseModeInRead which is EXCEPTION by default + The datetimeRebaseMode option allows to specify the rebasing mode for the values of the date, timestamp-micros, timestamp-millis logical types from the Julian to Proleptic Gregorian calendar.
+ Currently supported modes are: + + + read and function from_avro + ## Configuration @@ -317,6 +330,31 @@ Configuration of Avro can be done using the `setConf` method on SparkSession or 2.4.0 + + spark.sql.avro.datetimeRebaseModeInRead + EXCEPTION + The rebasing mode for the values of the date, timestamp-micros, timestamp-millis logical types from the Julian to Proleptic Gregorian calendar:
+ + This config is only effective if the writer info (like Spark, Hive) of the Avro files is unknown. + + 3.0.0 + + + spark.sql.avro.datetimeRebaseModeInWrite + EXCEPTION + The rebasing mode for the values of the date, timestamp-micros, timestamp-millis logical types from the Proleptic Gregorian to Julian calendar:
+ + + 3.0.0 + ## Compatibility with Databricks spark-avro diff --git a/docs/sql-data-sources-parquet.md b/docs/sql-data-sources-parquet.md index d34f8a698fac..119eae56ebf7 100644 --- a/docs/sql-data-sources-parquet.md +++ b/docs/sql-data-sources-parquet.md @@ -252,6 +252,42 @@ REFRESH TABLE my_table; +## Data Source Option + +Data source options of Parquet can be set via: +* the `.option`/`.options` methods of `DataFrameReader` or `DataFrameWriter` +* the `.option`/`.options` methods of `DataStreamReader` or `DataStreamWriter` + + + + + + + + + + + + + + + +
Property NameDefaultMeaningScope
datetimeRebaseModeThe SQL config spark.sql.parquet .datetimeRebaseModeInRead which is EXCEPTION by defaultThe datetimeRebaseMode option allows to specify the rebasing mode for the values of the DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS logical types from the Julian to Proleptic Gregorian calendar.
+ Currently supported modes are: +
    +
  • EXCEPTION: fails in reads of ancient dates/timestamps that are ambiguous between the two calendars.
    • +
  • CORRECTED: loads dates/timestamps without rebasing.
    • +
  • LEGACY: performs rebasing of ancient dates/timestamps from the Julian to Proleptic Gregorian calendar.
    • +
+ 		read
int96RebaseModeThe SQL config spark.sql.parquet .int96RebaseModeInRead which is EXCEPTION by defaultThe int96RebaseMode option allows to specify the rebasing mode for INT96 timestamps from the Julian to Proleptic Gregorian calendar.
+ Currently supported modes are: +
    +
  • EXCEPTION: fails in reads of ancient INT96 timestamps that are ambiguous between the two calendars.
    • +
  • CORRECTED: loads INT96 timestamps without rebasing.
    • +
  • LEGACY: performs rebasing of ancient timestamps from the Julian to Proleptic Gregorian calendar.
    • +
+ 		read
+ ### Configuration Configuration of Parquet can be done using the `setConf` method on `SparkSession` or by running @@ -329,4 +365,54 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession 1.6.0 + +spark.sql.parquet.datetimeRebaseModeInRead + EXCEPTION + The rebasing mode for the values of the DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS logical types from the Julian to Proleptic Gregorian calendar:
+ + This config is only effective if the writer info (like Spark, Hive) of the Parquet files is unknown. + + 3.0.0 + + + spark.sql.parquet.datetimeRebaseModeInWrite + EXCEPTION + The rebasing mode for the values of the DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS logical types from the Proleptic Gregorian to Julian calendar:
+ + + 3.0.0 + + + spark.sql.parquet.int96RebaseModeInRead + EXCEPTION + The rebasing mode for the values of the INT96 timestamp type from the Julian to Proleptic Gregorian calendar:
+ + This config is only effective if the writer info (like Spark, Hive) of the Parquet files is unknown. + + 3.1.0 + + + spark.sql.parquet.int96RebaseModeInWrite + EXCEPTION + The rebasing mode for the values of the INT96 timestamp type from the Proleptic Gregorian to Julian calendar:
+ + + 3.1.0 +