Skip to content

[SPARK-17075][SQL] implemented filter estimation #16395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ron8hu wants to merge 36 commits into from
Closed

[SPARK-17075][SQL] implemented filter estimation #16395

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
79ba742
implemented first version of filter estimation
ron8hu Dec 23, 2016
c46ccbf
added evaluateBinaryForNumeric
ron8hu Dec 29, 2016
cf42e5e
support all binary expressions and add test cases
ron8hu Dec 31, 2016
22f9637
Use AttributeMap for column statistics
ron8hu Jan 3, 2017
dfe8eb2
maintain an ExprId-to-Attribute map
ron8hu Jan 3, 2017
84d6033
fixed sql/test:scalastyle errors
ron8hu Jan 3, 2017
c5e3a6b
make mutableColStats start from empty every time estimate is called
ron8hu Jan 4, 2017
c6dcf90
make FilterEstimation a class
ron8hu Jan 9, 2017
3826bd0
move files to new directory statsEstimation
ron8hu Jan 10, 2017
f007a4d
added files to statsEstimation
ron8hu Jan 10, 2017
7caf600
reduce the dependency of unit test case FilterEstimationSuite
ron8hu Jan 10, 2017
6fe1994
make FilterEstimationSuite modular
ron8hu Jan 11, 2017
490f41a
use variable binding pattern matching
ron8hu Jan 12, 2017
4bc3008
change return type to Option[Double]
ron8hu Jan 13, 2017
7af19a6
use the unified computeStats method
ron8hu Jan 13, 2017
af911e1
make method calculateSingleCondition return Option[Double]
ron8hu Jan 14, 2017
2121ff2
add date and timestamp tests
ron8hu Jan 15, 2017
487813c
fix scalastyle error
ron8hu Jan 15, 2017
0f73034
add additional date / timestamp tests
ron8hu Jan 15, 2017
f24cf3e
update code based on wzhfy's comments
ron8hu Jan 16, 2017
3ebf3a8
filtered NDV should be no larger than initial NDV
ron8hu Jan 17, 2017
9763635
add tests to handle decimal data type
ron8hu Jan 18, 2017
35c213f
add test cases for float and double types
ron8hu Jan 18, 2017
6b8aab3
add cast-as-date test cases
ron8hu Jan 20, 2017
894d85c
update calls to getOutputSize
ron8hu Jan 20, 2017
97aacdf
use Range.isIntersected to decide if a literal is in boundary
ron8hu Feb 14, 2017
2b4a10a
handle date/timestamp string literal
ron8hu Feb 15, 2017
f54a6ce
solve merge conflict in EstimationUtils
ron8hu Feb 16, 2017
07e6320
remove useless type checking since typecocercion already did
ron8hu Feb 16, 2017
7ba6609
add string column tests. remove float time tests.
ron8hu Feb 18, 2017
1f3619f
improve readability
ron8hu Feb 18, 2017
6411d21
specify update = false for Or condition
ron8hu Feb 18, 2017
11b6a0b
remove the unused import. save with Unix style line ending
ron8hu Feb 19, 2017
298d255
update date column test case
ron8hu Feb 21, 2017
eac69af
clean up internal/external type conversion
ron8hu Feb 24, 2017
a48a4fd
use Javadoc style indentation for multiline comments
ron8hu Feb 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion ...st/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTypes}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical.statsEstimation.{AggregateEstimation, EstimationUtils, JoinEstimation, ProjectEstimation}
import org.apache.spark.sql.catalyst.plans.logical.statsEstimation._
import org.apache.spark.sql.types._
import org.apache.spark.util.Utils

Expand Down Expand Up @@ -129,6 +129,14 @@ case class Filter(condition: Expression, child: LogicalPlan)
.filterNot(SubqueryExpression.hasCorrelatedSubquery)
child.constraints.union(predicates.toSet)
}

override def computeStats(conf: CatalystConf): Statistics = {
if (conf.cboEnabled) {
FilterEstimation(this, conf).estimate.getOrElse(super.computeStats(conf))
} else {
super.computeStats(conf)
}
}
}

abstract class SetOperation(left: LogicalPlan, right: LogicalPlan) extends BinaryNode {
Expand Down
Loading