[SPARK-37705][SQL] Rebase timestamps in the session time zone saved in Parquet/Avro metadata #34973

MaxGekk · 2021-12-21T13:16:18Z

What changes were proposed in this pull request?

In the PR, I propose to add new metadata key org.apache.spark.timeZone which Spark writes to Parquet/Avro matadata while performing of datetimes rebase in the LEGACY mode (see the SQL configs:

spark.sql.parquet.datetimeRebaseModeInWrite,
spark.sql.parquet.int96RebaseModeInWrite and
spark.sql.avro.datetimeRebaseModeInWrite).

The writers uses the current session time zone (see the SQL config spark.sql.session.timeZone) in rebasing of Parquet/Avro timestamp columns.

At the reader side, Spark tries to get info about the writer's time zone from the new metadata property:

$ java -jar ~parquet-tools-1.12.0.jar meta ./part-00000-b0d90bf0-ce60-4b4f-b453-b33f61ab2b2a-c000.snappy.parquet
...
extra:       org.apache.spark.timeZone = America/Los_Angeles
extra:       org.apache.spark.legacyDateTime =

and use it in rebasing timestamps to the Proleptic Gregorian calendar. In the case when the reader cannot retrieve the original time zone from Parquet/Avro metadata, it uses the default JVM time zone for backward compatibility.

Why are the changes needed?

Before the changes, Spark assumes that a writer uses the default JVM time zone while rebasing of dates/timestamps. And if a reader and the writer have different JVM time zone settings, the reader cannot load such columns in the LEGACY mode correctly. So, the reader will have full info about writer settings after the changes.

Does this PR introduce any user-facing change?

Yes. After the changes, Parquet/Avro writers use the session time zone while timestamp rebasing in the LEGACY mode instead of the default JVM time zone. Need to highlight that the session time zone is set to the JVM time zone by default.

How was this patch tested?

By running new tests:

$ build/sbt "test:testOnly *ParquetRebaseDatetimeV1Suite"
$ build/sbt "test:testOnly *ParquetRebaseDatetimeV2Suite"
$ build/sbt "test:testOnly *AvroV1Suite"
$ build/sbt "test:testOnly *AvroV2Suite"

And related existing test suites:

$ build/sbt "test:testOnly *DateTimeUtilsSuite"
$ build/sbt "test:testOnly *RebaseDateTimeSuite"
$ build/sbt "test:testOnly *TimestampFormatterSuite"
$ build/sbt "avro/test:testOnly org.apache.spark.sql.avro.AvroCatalystDataConversionSuite"
$ build/sbt "test:testOnly *AvroRowReaderSuite"
$ build/sbt "test:testOnly *AvroSerdeSuite"
$ build/sbt "test:testOnly *ParquetVectorizedSuite"

Also modified the test SPARK-31159: rebasing timestamps in write to check loading timestamps in the LEGACY mode when the session time zone and JVM time zone are different.
Generated parquet files by Spark 3.2.0 (the commit 5d45a41) using the test "SPARK-31806: generate test files for checking compatibility with Spark 2.4". The parquet files don't contain info about the original time zone:

$ java -jar ~/Downloads/parquet-tools-1.12.0.jar meta sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_0.snappy.parquet
file:        file:/Users/maximgekk/proj/parquet-rebase-save-tz/sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_0.snappy.parquet
creator:     parquet-mr version 1.12.1 (build 2a5c06c58fa987f85aa22170be14d927d5ff6e7d)
extra:       org.apache.spark.version = 3.2.0
extra:       org.apache.spark.legacyINT96 =
extra:       org.apache.spark.sql.parquet.row.metadata = {"type":"struct","fields":[{"name":"dict","type":"timestamp","nullable":true,"metadata":{}},{"name":"plain","type":"timestamp","nullable":true,"metadata":{}}]}
extra:       org.apache.spark.legacyDateTime =

file schema: spark_schema
--------------------------------------------------------------------------------
dict:        OPTIONAL INT64 L:TIMESTAMP(MICROS,true) R:0 D:1
plain:       OPTIONAL INT64 L:TIMESTAMP(MICROS,true) R:0 D:1

By running the test "SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps", check loading of mixed parquet files generated by Spark 2.4.5/2.4.6 and 3.2.0/master.

Generated avro files by Spark 3.2.0 (the commit 5d45a41) using the test "SPARK-31855: generate test files for checking compatibility with Spark 2.4". The avro files don't contain info about the original time zone:

$ java -jar ~/Downloads/avro-tools-1.9.2.jar getmeta external/avro/src/test/resources/before_1582_timestamp_micros_v3_2_0.avro
avro.schema	{"type":"record","name":"topLevelRecord","fields":[{"name":"dt","type":[{"type":"long","logicalType":"timestamp-micros"},"null"]}]}
org.apache.spark.version	3.2.0
avro.codec	snappy
org.apache.spark.legacyDateTime

By running the test "SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps", check loading of mixed avro files generated by Spark 2.4.5/2.4.6 and 3.2.0/master.

SparkQA · 2021-12-21T14:28:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50915/

…Micros

SparkQA · 2021-12-21T15:41:21Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50915/

…Micros

SparkQA · 2021-12-21T18:39:00Z

Test build #146440 has finished for PR 34973 at commit 7e3ee92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-22T10:32:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50950/

SparkQA · 2021-12-22T11:25:38Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50950/

SparkQA · 2021-12-22T13:39:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50957/

MaxGekk · 2021-12-22T14:31:32Z

@cloud-fan @sadikovi Could you have a look at this PR. I am still working on it but existing tests have passed already. Need to just add more tests for the cases when JVM and the session time zone are different in the LEGACY mode, and check backward compatibility (maybe something more, any ideas are welcome).

SparkQA · 2021-12-22T14:49:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50957/

SparkQA · 2021-12-22T14:49:24Z

Test build #146474 has finished for PR 34973 at commit e01a29f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-22T18:09:20Z

Test build #146481 has finished for PR 34973 at commit 64c39a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sadikovi · 2021-12-22T22:14:57Z

Thanks, I will take a look later. I noticed that we are changing OSS Spark method signature. Would it be possible to somehow avoid breaking binary compatibility between OSS classes/methods? At least, it would be good to assess the changes from this point.

SparkQA · 2021-12-23T23:04:18Z

Test build #146538 has finished for PR 34973 at commit 2a3fd19.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-24T06:41:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51026/

SparkQA · 2021-12-24T07:26:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51026/

SparkQA · 2021-12-24T10:46:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51036/

SparkQA · 2021-12-24T11:06:15Z

Test build #146551 has finished for PR 34973 at commit 6282a9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-24T11:48:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51036/

SparkQA · 2021-12-24T12:06:19Z

Test build #146561 has finished for PR 34973 at commit 68a3504.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-24T14:42:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51041/

SparkQA · 2021-12-24T15:43:56Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/51041/

SparkQA · 2021-12-24T19:17:17Z

Test build #146566 has finished for PR 34973 at commit e06f5f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-12-27T11:38:05Z

external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

      "2_4_5" -> failInRead _,
-      "2_4_6" -> successInRead _
+      "2_4_6" -> successInRead _,
+      "3_2_2" -> successInRead _


3.2.2? We have not released 3.2.1 yet.

I used branch-3.2 to generate the golden files. See the version in PR description:

$ java -jar parquet-tools-1.12.0.jar meta sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_2.snappy.parquet file: file:/Users/maximgekk/proj/parquet-rebase-save-tz/sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_2.snappy.parquet creator: parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94) extra: org.apache.spark.version = 3.2.2

I have checked out branch-3.2 and ran:

$ ./build/mvn -Phive -Phive-thriftserver -Dskip -DskipTests package

and got core/target/extra-resources/spark-version-info.properties:

$ cat ./core/target/extra-resources/spark-version-info.properties version=3.2.2-SNAPSHOT user=maximgekk revision=d1cd110c20817eb1ccd716e099be5712df1f670c branch=branch-3.2 date=2021-12-23T19:12:40Z url=https://github.com/apache/spark.git

should be 3.2.1-SNAPSHOT Not? cc @gengliangwang @cloud-fan ?

Let me take the commit 5d45a41 and generate the golden files using it.

Yes it should be 3.2.1-SNAPSHOT..The RC script aborted sometimes. This should be a mistake when I rerun the script.

@gengliangwang can we fix it? otherwise, we will have a problem when releasing 3.2.1.

cloud-fan · 2021-12-27T11:40:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala

+   *               in Proleptic Gregorian calendar. It can be negative.
+   * @return The rebased microseconds since the epoch in Julian calendar.
+   */
+  def rebaseGregorianToJulianMicros(micros: Long): Long = {


is it used in test only?

Not only, see

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala

Line 328 in f7dabd8

val julianMicros = rebaseGregorianToJulianMicros(timestamp)

cloud-fan · 2021-12-27T11:40:19Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala

+   *               in the Julian calendar. It can be negative.
+   * @return The rebased microseconds since the epoch in Proleptic Gregorian calendar.
+   */
+  def rebaseJulianToGregorianMicros(micros: Long): Long = {


Actually, it should be used everywhere except rebasing in Parquet/Avro like timestamp formatting.

how many places do we have? we should understand the places that rely on JVM timezone.

rebaseJulianToGregorianMicros is used from:

fromJavaTimestamp() which is used (55 times) from

CatalystTypeConverter.TimestampConverter

CatalogColumnStat.fromExternalString()

Literal.apply()

LegacyFastTimestampFormatter.format()

LegacySimpleTimestampFormatter.parse()

HiveInspectors

HadoopTableReader

KafkaRecordToRowConverter.toInternalRowWithoutHeaders & toInternalRowWithHeaders

JdbcUtils.makeGetter()

OrcAtomicColumnVector

ParquetFilters.timestampToMicros <-- NEED TO RE-CHECK THIS ONE

In many tests

LegacyFastTimestampFormatter.parse() which is used (20 times) from:

CSVInferSchema

UnivocityGenerator

UnivocityParser

JacksonGenerator

JacksonParser

JsonInferSchema

rebaseGregorianToJulianMicros is used from:

toJavaTimestamp which is used (33 times) from:

CatalystTypeConverter.TimestampConverter

Iso8601TimestampFormatter.format()

LegacySimpleTimestampFormatter.format

HiveInspectors

BaseScriptTransformationExec.outputFieldWriters

JdbcUtils.makeSetter

OrcFilters.castLiteralValue

OrcSerializer.newConverter

LegacyFastTimestampFormatter.format() (the same as for parse() above).

Then we are fine:

when converting between java.sql.Timestamp, we must rely on JVM timezone

when reading/writing JSON/CSV with legacy mode, we need to follow the legacy behavior and rebase the timestamp. We should have used the session timezone to rebase, but the ship is already sailed and we have to keep using JVM timezone now.

ParquetFilters.timestampToMicros <-- NEED TO RE-CHECK THIS ONE

This is fine too since the rebasing happens locally on the same JVM. Also users can enable Java 8 for filters, see #28696

cloud-fan · 2021-12-27T11:42:28Z

...ain/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedPlainValuesReader.java

        for (int i = 0; i < total; i += 1) {
-          c.putLong(rowId + i, RebaseDateTime.rebaseJulianToGregorianMicros(buffer.getLong()));
+          c.putLong(
+              rowId + i,


nit: 2 spaces indentation

cloud-fan · 2021-12-27T11:50:58Z

external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

-  // "SPARK-31183: compatibility with Spark 2.4 in reading dates/timestamps"
-  ignore("SPARK-31855: generate test files for checking compatibility with Spark 2.4") {
+  // "SPARK-31183, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps"
+  ignore("SPARK-31855: generate test files for checking compatibility with Spark 2.4/3.2") {


how do you use this test to generate the new testing data files?

Just replace ignore -> test, and run it. For example, I checkout branch-3.2 and ran the test to generate the golden files. After that, I copied new files to PR's branch.

hmm, looking at the code, seems the version is fixed as 2_4_6

Yep, made it more flexible by using SPARK_VERSION_SHORT

cloud-fan · 2021-12-27T11:53:57Z

...st/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala

      "2_4_5" -> failInRead _,
-      "2_4_6" -> successInRead _).foreach { case (version, checkDefaultRead) =>
+      "2_4_6" -> successInRead _,
+      "3_2_2" -> successInRead _).foreach { case (version, checkDefaultRead) =>


ditto, should be 3_2_1?

The generated files have 3.2.2 in metadata, in fact. I would keep it as is. For us, more important that the files have the keys:

extra: org.apache.spark.legacyINT96 = extra: org.apache.spark.legacyDateTime =

but not any time zone.

cloud-fan · 2021-12-28T13:40:04Z

thanks, merging to master!

cloud-fan · 2021-12-28T13:40:19Z

@MaxGekk can you open a backport PR for 3.2? There are conflicts.

…n Parquet/Avro metadata In the PR, I propose to add new metadata key `org.apache.spark.timeZone` which Spark writes to Parquet/Avro matadata while performing of datetimes rebase in the `LEGACY` mode (see the SQL configs: - `spark.sql.parquet.datetimeRebaseModeInWrite`, - `spark.sql.parquet.int96RebaseModeInWrite` and - `spark.sql.avro.datetimeRebaseModeInWrite`). The writers uses the current session time zone (see the SQL config `spark.sql.session.timeZone`) in rebasing of Parquet/Avro timestamp columns. At the reader side, Spark tries to get info about the writer's time zone from the new metadata property: ``` $ java -jar ~parquet-tools-1.12.0.jar meta ./part-00000-b0d90bf0-ce60-4b4f-b453-b33f61ab2b2a-c000.snappy.parquet ... extra: org.apache.spark.timeZone = America/Los_Angeles extra: org.apache.spark.legacyDateTime = ``` and use it in rebasing timestamps to the Proleptic Gregorian calendar. In the case when the reader cannot retrieve the original time zone from Parquet/Avro metadata, it uses the default JVM time zone for backward compatibility. Before the changes, Spark assumes that a writer uses the default JVM time zone while rebasing of dates/timestamps. And if a reader and the writer have different JVM time zone settings, the reader cannot load such columns in the `LEGACY` mode correctly. So, the reader will have full info about writer settings after the changes. Yes. After the changes, Parquet/Avro writers use the session time zone while timestamp rebasing in the `LEGACY` mode instead of the default JVM time zone. Need to highlight that the session time zone is set to the JVM time zone by default. 1. By running new tests: ``` $ build/sbt "test:testOnly *ParquetRebaseDatetimeV1Suite" $ build/sbt "test:testOnly *ParquetRebaseDatetimeV2Suite" $ build/sbt "test:testOnly *AvroV1Suite" $ build/sbt "test:testOnly *AvroV2Suite" ``` 2. And related existing test suites: ``` $ build/sbt "test:testOnly *DateTimeUtilsSuite" $ build/sbt "test:testOnly *RebaseDateTimeSuite" $ build/sbt "test:testOnly *TimestampFormatterSuite" $ build/sbt "avro/test:testOnly org.apache.spark.sql.avro.AvroCatalystDataConversionSuite" $ build/sbt "test:testOnly *AvroRowReaderSuite" $ build/sbt "test:testOnly *AvroSerdeSuite" $ build/sbt "test:testOnly *ParquetVectorizedSuite" ``` 3. Also modified the test `SPARK-31159: rebasing timestamps in write` to check loading timestamps in the LEGACY mode when the session time zone and JVM time zone are different. 4. Generated parquet files by Spark 3.2.0 (the commit apache@5d45a41) using the test `"SPARK-31806: generate test files for checking compatibility with Spark 2.4"`. The parquet files don't contain info about the original time zone: ``` $ java -jar ~/Downloads/parquet-tools-1.12.0.jar meta sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_0.snappy.parquet file: file:/Users/maximgekk/proj/parquet-rebase-save-tz/sql/core/src/test/resources/test-data/before_1582_timestamp_micros_v3_2_0.snappy.parquet creator: parquet-mr version 1.12.1 (build 2a5c06c58fa987f85aa22170be14d927d5ff6e7d) extra: org.apache.spark.version = 3.2.0 extra: org.apache.spark.legacyINT96 = extra: org.apache.spark.sql.parquet.row.metadata = {"type":"struct","fields":[{"name":"dict","type":"timestamp","nullable":true,"metadata":{}},{"name":"plain","type":"timestamp","nullable":true,"metadata":{}}]} extra: org.apache.spark.legacyDateTime = file schema: spark_schema -------------------------------------------------------------------------------- dict: OPTIONAL INT64 L:TIMESTAMP(MICROS,true) R:0 D:1 plain: OPTIONAL INT64 L:TIMESTAMP(MICROS,true) R:0 D:1 ``` By running the test `"SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps"`, check loading of mixed parquet files generated by Spark 2.4.5/2.4.6 and 3.2.0/master. 5. Generated avro files by Spark 3.2.0 (the commit apache@5d45a41) using the test `"SPARK-31855: generate test files for checking compatibility with Spark 2.4"`. The avro files don't contain info about the original time zone: ``` $ java -jar ~/Downloads/avro-tools-1.9.2.jar getmeta external/avro/src/test/resources/before_1582_timestamp_micros_v3_2_0.avro avro.schema {"type":"record","name":"topLevelRecord","fields":[{"name":"dt","type":[{"type":"long","logicalType":"timestamp-micros"},"null"]}]} org.apache.spark.version 3.2.0 avro.codec snappy org.apache.spark.legacyDateTime ``` By running the test `"SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps"`, check loading of mixed avro files generated by Spark 2.4.5/2.4.6 and 3.2.0/master. Closes apache#34973 from MaxGekk/parquet-rebase-save-tz. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit ef3a470) Signed-off-by: Max Gekk <[email protected]>

MaxGekk added 2 commits December 21, 2021 16:07

Write the metadata key 'org.apache.spark.timeZone' to parquet files

158e6cb

Update tests

7e3ee92

github-actions bot added the SQL label Dec 21, 2021

Pass time zone id to the optimized version of rebaseJulianToGregorian…

c6d3a64

…Micros

Pass time zone id to the optimized version of rebaseGregorianToJulian…

8ec7883

…Micros

MaxGekk added 9 commits December 21, 2021 22:55

Use the session time zone in timestamp rebasing

b510ff1

Merge remote-tracking branch 'origin/master' into parquet-rebase-save-tz

609717c

Write the session time zone to Avro metadata

c440169

SPARK_LEGACY_DATETIME -> SPARK_LEGACY_DATETIME_METADATA_KEY

37dc837

SPARK_LEGACY_INT96 -> SPARK_LEGACY_INT96_METADATA_KEY

08d4f78

Extract common code to getRebaseMode()

70bcd62

Add RebaseSpec

8d186ad

getRebaseMode -> getRebaseSpec

3a28d07

Use saved time zone in loading

e01a29f

github-actions bot added the AVRO label Dec 22, 2021

MaxGekk changed the title ~~[WIP][SPARK-37705][SQL] Write the session time zone in Parquet file metadata~~ [WIP][SPARK-37705][SQL] Write the session time zone in Parquet/Avro file metadata Dec 22, 2021

MaxGekk added 2 commits December 22, 2021 15:27

Change the Parquet Vectorized reader

21e688b

Make Java style checker happy

64c39a7

Modified a test to check JVM and session time zones

ece9afc

Add parquet files

6282a9e

MaxGekk added 2 commits December 24, 2021 12:32

Check rebasing microseconds/milliseconds in Avro

605f350

Check backward compatibility in Avro

68a3504

Fix comments

e06f5f4

cloud-fan reviewed Dec 27, 2021

View reviewed changes

MaxGekk added 5 commits December 27, 2021 20:39

Fix indentations

0f77adb

Merge remote-tracking branch 'origin/master' into parquet-rebase-save-tz

12a894a

Re-gen Avro golden files by Spark 3.2.0

8eebcf8

Re-gen Parquet golden files by Spark 3.2.0

7b734e4

Use SPARK_VERSION_SHORT in generating golden files

5b7de02

cloud-fan closed this in ef3a470 Dec 28, 2021

ttnghia mentioned this pull request Nov 3, 2023

[FEA] Take into account org.apache.spark.timeZone in Parquet/Avro from Spark 3.2 NVIDIA/spark-rapids#9632

Closed

[SPARK-37705][SQL] Rebase timestamps in the session time zone saved in Parquet/Avro metadata #34973

[SPARK-37705][SQL] Rebase timestamps in the session time zone saved in Parquet/Avro metadata #34973

Uh oh!

Conversation

MaxGekk commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

MaxGekk commented Dec 22, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

SparkQA commented Dec 22, 2021

Uh oh!

sadikovi commented Dec 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 23, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

SparkQA commented Dec 24, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Dec 21, 2021 •

edited

Loading

sadikovi commented Dec 22, 2021 •

edited

Loading

cloud-fan Dec 27, 2021 •

edited

Loading