Skip to content

Conversation

@myandpr
Copy link
Contributor

@myandpr myandpr commented Nov 3, 2021

What problem is solved by this MR?Why this MR is needed?

It doesn`t work when I use 'date_sub(current_date(), 10)' in 'spark_catalog.system.expire_snapshots'.

because we use a parser extension, the procedures do not allow arbitrary expressions.

I think that we'd better modify the document first to prevent users from encountering errors.

environment:

Spark Iceberg
3.0 0.12
scala> spark.sql("CALL spark_catalog.system.expire_snapshots('db.table', date_sub(current_date(), 10), 5)")
org.apache.spark.sql.catalyst.parser.extensions.IcebergParseException:
no viable alternative at input 'date_sub('(line 1, pos 81)

== SQL ==
CALL spark_catalog.system.expire_snapshots('db.table', date_sub(current_date(), 10), 5)
---------------------------------------------------------------------------------^^^

  at org.apache.spark.sql.catalyst.parser.extensions.IcebergParseException.withCommand(IcebergSparkSqlExtensionsParser.scala:294)
  at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parse(IcebergSparkSqlExtensionsParser.scala:158)
  at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:105)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:605)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:605)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
  ... 47 elided

How to solve this problem?

I think that we'd better modify the document first to prevent users from encountering errors.

Related issue

#2341

@github-actions github-actions bot added the docs label Nov 3, 2021
@myandpr
Copy link
Contributor Author

myandpr commented Nov 3, 2021

@RussellSpitzer , could you help to review this PR? :)

Copy link
Member

@RussellSpitzer RussellSpitzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I have been meaning to fix this for a while

@RussellSpitzer RussellSpitzer merged commit e7c8fc7 into apache:master Nov 3, 2021
KnightChess pushed a commit to KnightChess/iceberg that referenced this pull request Nov 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants