[SPARK-21180][SQL] Remove conf from stats functions since now we have conf in LogicalPlan #18391
Conversation
|
cc @rxin |
| } | ||
|
|
||
| override def computeStats(conf: SQLConf): Statistics = | ||
| override def computeStats: Statistics = |
There was a problem hiding this comment.
Can we remove import org.apache.spark.sql.internal.SQLConf, too?
There was a problem hiding this comment.
I checked all modified files and removed unused imports. Thanks!
| )) | ||
|
|
||
| override def computeStats(conf: SQLConf): Statistics = { | ||
| override def computeStats: Statistics = { |
There was a problem hiding this comment.
Can we remove import org.apache.spark.sql.internal.SQLConf, too?
| */ | ||
| final def stats(conf: SQLConf): Statistics = statsCache.getOrElse { | ||
| statsCache = Some(computeStats(conf)) | ||
| final def stats: Statistics = statsCache.getOrElse { |
|
|
||
| override def computeStats(conf: SQLConf): Statistics = { | ||
| val stats = child.stats(conf) | ||
| override def computeStats: Statistics = { |
|
Test build #78459 has finished for PR 18391 at commit
|
|
thanks, SGTM |
|
Test build #78496 has finished for PR 18391 at commit
|
| )) | ||
|
|
||
| override def computeStats(conf: SQLConf): Statistics = { | ||
| override def computeStats: Statistics = { |
There was a problem hiding this comment.
Yes, this file's import org.apache.spark.sql.internal.SQLConf need to be removed
There was a problem hiding this comment.
Thanks! I checked the other interface file, there are several files with this name...
| SQLConf.get.setConf(SQLConf.CBO_ENABLED, false) | ||
| assert(plan.stats == expectedStatsCboOff) | ||
| } finally { | ||
| SQLConf.get.unsetConf(SQLConf.CBO_ENABLED) |
There was a problem hiding this comment.
first, get the original and then recover the original one in the finally block. You can check how we did it in SQLConfSuite
| // From UnaryNode.computeStats, childSize * outputRowSize / childRowSize | ||
| Statistics(sizeInBytes = 48 * (8 + 4 + 8) / (8 + 4))) | ||
| } finally { | ||
| SQLConf.get.unsetConf(SQLConf.CBO_ENABLED) |
| } | ||
|
|
||
| override def afterAll(): Unit = { | ||
| SQLConf.get.unsetConf(SQLConf.CBO_ENABLED) |
|
Test build #78521 has finished for PR 18391 at commit
|
|
LGTM |
|
Thanks! Merging to master. |
… conf in LogicalPlan ## What changes were proposed in this pull request? After wiring `SQLConf` in logical plan ([PR 18299](apache#18299)), we can remove the need of passing `conf` into `def stats` and `def computeStats`. ## How was this patch tested? Covered by existing tests, plus some modified existing tests. Author: wangzhenhua <[email protected]> Author: Zhenhua Wang <[email protected]> Closes apache#18391 from wzhfy/removeConf.
4 participants
What changes were proposed in this pull request?
After wiring
SQLConfin logical plan (PR 18299), we can remove the need of passingconfintodef statsanddef computeStats.How was this patch tested?
Covered by existing tests, plus some modified existing tests.