Skip to content

Conversation

@rxin
Copy link
Contributor

@rxin rxin commented Oct 13, 2016

What changes were proposed in this pull request?

This patch graduates a list of Spark SQL APIs and mark them stable.

The following are marked stable:

Dataset/DataFrame

  • functions, since 1.3
  • ColumnName, since 1.3
  • DataFrameNaFunctions, since 1.3.1
  • DataFrameStatFunctions, since 1.4
  • UserDefinedFunction, since 1.3
  • UserDefinedAggregateFunction, since 1.5
  • Window and WindowSpec, since 1.4

Data sources:

  • DataSourceRegister, since 1.5
  • RelationProvider, since 1.3
  • SchemaRelationProvider, since 1.3
  • CreatableRelationProvider, since 1.3
  • BaseRelation, since 1.3
  • TableScan, since 1.3
  • PrunedScan, since 1.3
  • PrunedFilteredScan, since 1.3
  • InsertableRelation, since 1.3

The following are kept experimental / evolving:

Data sources:

  • CatalystScan (tied to internal logical plans so it is not stable by definition)

Structured streaming:

  • all classes (introduced new in 2.0 and will likely change)

Dataset typed operations (introduced in 1.6 and 2.0 and might change, although probability is low)

  • all typed methods on Dataset
  • KeyValueGroupedDataset
  • o.a.s.sql.expressions.javalang.typed
  • o.a.s.sql.expressions.scalalang.typed
  • methods that return typed Dataset in SparkSession

We should discuss more whether we want to mark Dataset typed operations stable in 2.1.

How was this patch tested?

N/A - just annotation changes.

@rxin rxin changed the title Graduate a list of Spark SQL APIs to stable [SPARK-17900][SQL] Graduate a list of Spark SQL APIs to stable Oct 13, 2016
@SparkQA
Copy link

SparkQA commented Oct 13, 2016

Test build #66910 has finished for PR 15469 at commit f712dd4.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 13, 2016

Test build #3338 has finished for PR 15469 at commit f712dd4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor Author

rxin commented Oct 14, 2016

cc @marmbrus

@marmbrus
Copy link
Contributor

LGTM, merging to master.

@asfgit asfgit closed this in 72adfbf Oct 14, 2016
robert3005 pushed a commit to palantir/spark that referenced this pull request Nov 1, 2016
## What changes were proposed in this pull request?
This patch graduates a list of Spark SQL APIs and mark them stable.

The following are marked stable:

Dataset/DataFrame
- functions, since 1.3
- ColumnName, since 1.3
- DataFrameNaFunctions, since 1.3.1
- DataFrameStatFunctions, since 1.4
- UserDefinedFunction, since 1.3
- UserDefinedAggregateFunction, since 1.5
- Window and WindowSpec, since 1.4

Data sources:
- DataSourceRegister, since 1.5
- RelationProvider, since 1.3
- SchemaRelationProvider, since 1.3
- CreatableRelationProvider, since 1.3
- BaseRelation, since 1.3
- TableScan, since 1.3
- PrunedScan, since 1.3
- PrunedFilteredScan, since 1.3
- InsertableRelation, since 1.3

The following are kept experimental / evolving:

Data sources:
- CatalystScan (tied to internal logical plans so it is not stable by definition)

Structured streaming:
- all classes (introduced new in 2.0 and will likely change)

Dataset typed operations (introduced in 1.6 and 2.0 and might change, although probability is low)
- all typed methods on Dataset
- KeyValueGroupedDataset
- o.a.s.sql.expressions.javalang.typed
- o.a.s.sql.expressions.scalalang.typed
- methods that return typed Dataset in SparkSession

We should discuss more whether we want to mark Dataset typed operations stable in 2.1.

## How was this patch tested?
N/A - just annotation changes.

Author: Reynold Xin <[email protected]>

Closes apache#15469 from rxin/SPARK-17900.
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
## What changes were proposed in this pull request?
This patch graduates a list of Spark SQL APIs and mark them stable.

The following are marked stable:

Dataset/DataFrame
- functions, since 1.3
- ColumnName, since 1.3
- DataFrameNaFunctions, since 1.3.1
- DataFrameStatFunctions, since 1.4
- UserDefinedFunction, since 1.3
- UserDefinedAggregateFunction, since 1.5
- Window and WindowSpec, since 1.4

Data sources:
- DataSourceRegister, since 1.5
- RelationProvider, since 1.3
- SchemaRelationProvider, since 1.3
- CreatableRelationProvider, since 1.3
- BaseRelation, since 1.3
- TableScan, since 1.3
- PrunedScan, since 1.3
- PrunedFilteredScan, since 1.3
- InsertableRelation, since 1.3

The following are kept experimental / evolving:

Data sources:
- CatalystScan (tied to internal logical plans so it is not stable by definition)

Structured streaming:
- all classes (introduced new in 2.0 and will likely change)

Dataset typed operations (introduced in 1.6 and 2.0 and might change, although probability is low)
- all typed methods on Dataset
- KeyValueGroupedDataset
- o.a.s.sql.expressions.javalang.typed
- o.a.s.sql.expressions.scalalang.typed
- methods that return typed Dataset in SparkSession

We should discuss more whether we want to mark Dataset typed operations stable in 2.1.

## How was this patch tested?
N/A - just annotation changes.

Author: Reynold Xin <[email protected]>

Closes apache#15469 from rxin/SPARK-17900.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants