write.wap.enabled / spark.wap.branch / spark.wap.id behavior isn't really documented

[wimlewis-amazon]

Feature Request / Improvement

It's unclear from reading the doc what the effect of setting write.wap.enabled actually is. I can make some assumptions based on the write-audit-publish example, but those assumptions are almost certainly incomplete (for example, there are edge cases and failure conditions that I would need to understand before applying this in production, but the example only shows the "happy path".). As a user of the Iceberg libraries, I would like the documentation to be more explicit about what changes if I set either of these configuration options.

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[rodrigc]

I also did not fully understand write.wap.enabled.
Looking at the code, it seems like an additional configuration knob
that can turn off WAP even if spark.wap.branch is set:

iceberg/spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java

Line 123 in 17bda20

table.properties(), TableProperties.WRITE_AUDIT_PUBLISH_ENABLED, false);

iceberg/core/src/main/java/org/apache/iceberg/TableProperties.java

Line 310 in 17bda20

public static final String WRITE_AUDIT_PUBLISH_ENABLED = "write.wap.enabled";

[wimlewis-amazon]

Hmm, from those code snippets, it looks like enabled is a table property which enables the behavior, and branch is a spark session property that indicates which branch to read (and alternatively disables WAP if it's unset). That's a little weird (shouldn't at least one of those be a write options property? And it seems that WAP affects the behavior of table read as well, which isn't what I would naively expect, although from the PR discussions I can kind of see the idea.)

Digging through git-blame, I found a bunch of discussion, in PRs #6651, #6965, which references these behaviors but it's still not clear what the final intended behavior is — what I should expect as a user of the spark api.

[rodrigc]

@wimlewis-amazon my understanding is that:

1. write.wap.enabled is a table property, and must be set on the table in order to do WAP
2. spark.wap.branch is a Spark sessionproperty that specifies the name of the branch

Looking at the code, it looks like you need to have the table property and the Spark session property set in order
to do WAP.
This lines up with the example in the docs here: https://iceberg.apache.org/docs/latest/branching/#audit-branch

Originally I looked at the example in the docs, and didn't really understand why there was a table property
and a Spark session property for WAP. Looking at the code snippets above improved my understanding.
But I admit, I find this a bit confusing as an end-user.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[wimlewis-amazon]

This is still a pretty relevant issue.

[Dima1224]

I agree the documentation is lacking in this regard. Specifically, I am trying to understand:

• Can I write to a table which is write.wap.enabled=true in the standard ways and expect standard results?
• What exactly is the effect of setting write.wap.enabled=true and why is this flag necessary?

I found this thread helpful, but it would be nice to see this info in the docs.

[gphilipp]

I'm a bit confused too. I've set spark.wap.id to some value. Now if I set spark.wap.branch as well, it fails with:

org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.iceberg.exceptions.ValidationException: Cannot set both WAP ID and branch

It seems they’re mutually exclusive. What’s the intended difference between spark.wap.id and spark.wap.branch? When should each be used?