[SPARK-40708][SQL] Auto update table statistics based on write metrics #38496
Conversation
wankunde
changed the title
wankunde
changed the title
|
cc @wangyum |
There was a problem hiding this comment.
Hm, we should consider about partition Statistics here. If we overwrite the part of the partitions, it would get wrong table statistcs.
There was a problem hiding this comment.
Hi, @jackylee-ch Thanks for your review. It seems we can only update stats for overwriting non-partition table.
There was a problem hiding this comment.
hmm... so for overwriting non-partition table, if autoSizeUpdateEnabled is true, we cannot use wroteStats to update statistics?
There was a problem hiding this comment.
@LuciferYang Good idea, update the code, prefer to use wrote stats to update non-partition table statistics if possible.
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
Outdated
Show resolved
Hide resolved
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
Outdated
Show resolved
Hide resolved
wankunde
changed the title
wankunde
force-pushed
the
writeStats
branch
3 times, most recently
from
b646993 to
cd3d69d
Compare
|
Retest this please |
wankunde
force-pushed
the
writeStats
branch
2 times, most recently
from
1f17cb7 to
b0ed310
Compare
wankunde
requested review from
LuciferYang and
jackylee-ch
and removed request for
LuciferYang and
jackylee-ch
|
Support partition statistics? |
@melin I'm working on the supporting of partition statistics update, it relies on workers to return detailed partition statistics. |
Can consider the table or partition statistics released, the user can listen to these statistics, convenient display, before the magic code to obtain statistics, not very standard. |
|
Hi, @jackylee-ch @melin any update ? |
|
@LuciferYang @jackylee-ch Could you help to review this PR again? |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Is it done? |
5 participants
What changes were proposed in this pull request?
Update table size and rowCount based on spark write metrics
Why are the changes needed?
Auto update table stats after write job finished.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Add UT