[SPARK-41524][SS] Differentiate SQLConf and extraOptions in StateStoreConf for its usage in RocksDBConf#39069
[SPARK-41524][SS] Differentiate SQLConf and extraOptions in StateStoreConf for its usage in RocksDBConf#39069huanliwang-db wants to merge 4 commits intoapache:masterfrom
Conversation
…eConf for its usage in RocksDBConf
| private case class ConfEntry(name: String, default: String) { | ||
| def fullName: String = s"$ROCKSDB_CONF_NAME_PREFIX.${name}".toLowerCase(Locale.ROOT) | ||
| def sqlConfFullName: String = s"$ROCKSDB_SQL_CONF_NAME_PREFIX.${name}".toLowerCase(Locale.ROOT) | ||
| def lowerCaseName: String = name.toLowerCase(Locale.ROOT) |
There was a problem hiding this comment.
What about having different class for each case instead? It would be clearer to distinguish configs, e.g. we don't expect the same config to be set from both SQL conf and extraOptions. If we allow such case, it'll be a bit more complicated than what you proposed, e.g. we will have to define preference.
There was a problem hiding this comment.
After the change, the method name does not need additional FromSQL or FromExtraOptions since parameter will tell the fact.
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala
Outdated
Show resolved
Hide resolved
HeartSaVioR
changed the title
HeartSaVioR
left a comment
HeartSaVioR
left a comment
There was a problem hiding this comment.
Thanks for the work! Only one comment.
| val ROCKSDB_CONF_NAME_PREFIX = "spark.sql.streaming.stateStore.rocksdb" | ||
| val ROCKSDB_SQL_CONF_NAME_PREFIX = "spark.sql.streaming.stateStore.rocksdb" | ||
|
|
||
| private case class ConfEntry(name: String, default: String) { |
There was a problem hiding this comment.
abstract class or trait. extending case class is a bad practice for Scala.
EDIT: It's not just a bad practice. It's disallowed.
|
Thanks! Merging to master. |
|
Can one of the admins verify this patch? |
3 participants
What changes were proposed in this pull request?
Currently the usage of StateStoreConf is via confs, which composes both SQL confs and extraOptions into one. The name of config for extraOptions shouldn't have to follow the name prefix of SQL conf, because it's not bound to the context of SQL conf.
Why are the changes needed?
After differentiate SQL conf and extraOptions in StateStoreConf, we should be able to adopt more use case on operator level configs by using the extraOptions.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing UT should cover the change