Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,23 @@
package org.apache.spark.sql.sources.v2;

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.sources.v2.reader.DataSourceV2Reader;
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;

/**
* A mix-in interface for {@link DataSourceV2}. Data sources can implement this interface to
* provide data reading ability and scan the data from the data source.
*/
@InterfaceStability.Evolving
public interface ReadSupport {
public interface ReadSupport extends DataSourceV2 {

/**
* Creates a {@link DataSourceV2Reader} to scan the data from this data source.
* Creates a {@link DataSourceReader} to scan the data from this data source.
*
* If this method fails (by throwing an exception), the action would fail and no Spark job was
* submitted.
*
* @param options the options for the returned data source reader, which is an immutable
* case-insensitive string-to-string map.
*/
DataSourceV2Reader createReader(DataSourceV2Options options);
DataSourceReader createReader(DataSourceV2Options options);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not rename options as well?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on @rdblue's comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, let me rename it too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we only need V2 in the root interface: DataSourceV2

}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.spark.sql.sources.v2;

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.sources.v2.reader.DataSourceV2Reader;
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
import org.apache.spark.sql.types.StructType;

/**
Expand All @@ -30,10 +30,10 @@
* supports both schema inference and user-specified schema.
*/
@InterfaceStability.Evolving
public interface ReadSupportWithSchema {
public interface ReadSupportWithSchema extends DataSourceV2 {

/**
* Create a {@link DataSourceV2Reader} to scan the data from this data source.
* Create a {@link DataSourceReader} to scan the data from this data source.
*
* If this method fails (by throwing an exception), the action would fail and no Spark job was
* submitted.
Expand All @@ -45,5 +45,5 @@ public interface ReadSupportWithSchema {
* @param options the options for the returned data source reader, which is an immutable
* case-insensitive string-to-string map.
*/
DataSourceV2Reader createReader(StructType schema, DataSourceV2Options options);
DataSourceReader createReader(StructType schema, DataSourceV2Options options);
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
* session.
*/
@InterfaceStability.Evolving
public interface SessionConfigSupport {
public interface SessionConfigSupport extends DataSourceV2 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to extend DataSourceV2? Why add this in a commit that appears to be nothing more than a rename?

@dongjoon-hyun dongjoon-hyun Jan 29, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for @rdblue 's question.
@cloud-fan . Could you add more explanation about this change at PR description? According to the following, do we need to update Apache JIRA issue types from IMPROVEMENT to BUG?

Also this PR fixes some places that the mix-in interface doesn't extend the interface it aimed to mix in.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something we left behind. For a mix-in interface, it should extend the interface it aimed to mix in, so that we can guarantee, for example, classes implement SessionConfigSupport must also implement DataSourceV2.

We already did it in some mix-in interfaces, e.g. the streaming read support, SupportsScanUnsafeRow, SupportsScanColumnarBatch, etc. In this PR I just wanna fix the remaining ones, to make them consistent.

If you feel strongly about it, I can create a JIRA ticket for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's best to keep commits clean and focused. I'd say create a new JIRA for it and do all of the mix-ins at once.

+1 when this is separated to its own PR. Thanks!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created https://issues.apache.org/jira/browse/SPARK-23262 .

I'd like to keep the changes in this PR, as it's just several lines. It's not ideal but we are rushing for the 2.3 release, I wanna speed this up a little, what do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mixing large migration commits like this one with unrelated changes makes it harder to pick or revert changes without unintended side-effects. What happens if we realize that this rename was a bad idea? Reverting this commit would also revert the constraint that SessionConfigSupport extends DataSourceV2. Similarly, if we realize that these mix-ins don't need to extend DataSourceV2, then we would have to find and remove them all instead of reverting a commit. That might even sound okay, but when you're picking commits deliberately to patch branches, you need to make as few changes as possible and cherry-pick conflicts make that much harder.

The fact that you're rushing to get commits into 2.3 is even more concerning and reason to be careful, not a reason to relax our standards. Please move this to its own PR and fix all of the interfaces at once.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping me on the new PR. I'm happy to review it (though it is non-binding).


/**
* Key prefix of the session configs to propagate. Spark will extract all session configs that
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,32 +21,32 @@

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.sources.v2.writer.DataSourceV2Writer;
import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.types.StructType;

/**
* A mix-in interface for {@link DataSourceV2}. Data sources can implement this interface to
* provide data writing ability and save the data to the data source.
*/
@InterfaceStability.Evolving
public interface WriteSupport {
public interface WriteSupport extends DataSourceV2 {

/**
* Creates an optional {@link DataSourceV2Writer} to save the data to this data source. Data
* Creates an optional {@link DataSourceWriter} to save the data to this data source. Data
* sources can return None if there is no writing needed to be done according to the save mode.
*
* If this method fails (by throwing an exception), the action would fail and no Spark job was
* submitted.
*
* @param jobId A unique string for the writing job. It's possible that there are many writing
* jobs running at the same time, and the returned {@link DataSourceV2Writer} can
* jobs running at the same time, and the returned {@link DataSourceWriter} can
* use this job id to distinguish itself from other jobs.
* @param schema the schema of the data to be written.
* @param mode the save mode which determines what to do when the data are already in this data
* source, please refer to {@link SaveMode} for more details.
* @param options the options for the returned data source writer, which is an immutable
* case-insensitive string-to-string map.
*/
Optional<DataSourceV2Writer> createWriter(
Optional<DataSourceWriter> createWriter(
String jobId, StructType schema, SaveMode mode, DataSourceV2Options options);
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.apache.spark.annotation.InterfaceStability;

/**
* A reader factory returned by {@link DataSourceV2Reader#createDataReaderFactories()} and is
* A reader factory returned by {@link DataSourceReader#createDataReaderFactories()} and is
* responsible for creating the actual data reader. The relationship between
* {@link DataReaderFactory} and {@link DataReader}
* is similar to the relationship between {@link Iterable} and {@link java.util.Iterator}.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
* issues the scan request and does the actual data reading.
*/
@InterfaceStability.Evolving
public interface DataSourceV2Reader {
public interface DataSourceReader {

/**
* Returns the actual schema of this data source reader, which may be different from the physical
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
import org.apache.spark.sql.catalyst.expressions.Expression;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to push down arbitrary expressions as predicates to the data source.
* This is an experimental and unstable interface as {@link Expression} is not public and may get
* changed in the future Spark versions.
Expand All @@ -31,7 +31,7 @@
* process this interface.
*/
@InterfaceStability.Unstable
public interface SupportsPushDownCatalystFilters {
public interface SupportsPushDownCatalystFilters extends DataSourceReader {

/**
* Pushes down filters, and returns unsupported filters.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@
import org.apache.spark.sql.sources.Filter;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to push down filters to the data source and reduce the size of the data to be read.
*
* Note that, if data source readers implement both this interface and
* {@link SupportsPushDownCatalystFilters}, Spark will ignore this interface and only process
* {@link SupportsPushDownCatalystFilters}.
*/
@InterfaceStability.Evolving
public interface SupportsPushDownFilters {
public interface SupportsPushDownFilters extends DataSourceReader {

/**
* Pushes down filters, and returns unsupported filters.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
import org.apache.spark.sql.types.StructType;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to push down required columns to the data source and only read these columns during
* scan to reduce the size of the data to be read.
*/
@InterfaceStability.Evolving
public interface SupportsPushDownRequiredColumns {
public interface SupportsPushDownRequiredColumns extends DataSourceReader {

/**
* Applies column pruning w.r.t. the given requiredSchema.
Expand All @@ -35,7 +35,7 @@ public interface SupportsPushDownRequiredColumns {
* also OK to do the pruning partially, e.g., a data source may not be able to prune nested
* fields, and only prune top-level columns.
*
* Note that, data source readers should update {@link DataSourceV2Reader#readSchema()} after
* Note that, data source readers should update {@link DataSourceReader#readSchema()} after
* applying column pruning.
*/
void pruneColumns(StructType requiredSchema);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@
import org.apache.spark.annotation.InterfaceStability;

/**
* A mix in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to report data partitioning and try to avoid shuffle at Spark side.
*/
@InterfaceStability.Evolving
public interface SupportsReportPartitioning {
public interface SupportsReportPartitioning extends DataSourceReader {

/**
* Returns the output data partitioning that this reader guarantees.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@
import org.apache.spark.annotation.InterfaceStability;

/**
* A mix in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to report statistics to Spark.
*/
@InterfaceStability.Evolving
public interface SupportsReportStatistics {
public interface SupportsReportStatistics extends DataSourceReader {

/**
* Returns the basic statistics of this data source.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,19 @@
import org.apache.spark.sql.vectorized.ColumnarBatch;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to output {@link ColumnarBatch} and make the scan faster.
*/
@InterfaceStability.Evolving
public interface SupportsScanColumnarBatch extends DataSourceV2Reader {
public interface SupportsScanColumnarBatch extends DataSourceReader {
@Override
default List<DataReaderFactory<Row>> createDataReaderFactories() {
throw new IllegalStateException(
"createDataReaderFactories not supported by default within SupportsScanColumnarBatch.");
}

/**
* Similar to {@link DataSourceV2Reader#createDataReaderFactories()}, but returns columnar data
* Similar to {@link DataSourceReader#createDataReaderFactories()}, but returns columnar data
* in batches.
*/
List<DataReaderFactory<ColumnarBatch>> createBatchDataReaderFactories();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,13 @@
import org.apache.spark.sql.catalyst.expressions.UnsafeRow;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to output {@link UnsafeRow} directly and avoid the row copy at Spark side.
* This is an experimental and unstable interface, as {@link UnsafeRow} is not public and may get
* changed in the future Spark versions.
*/
@InterfaceStability.Unstable
public interface SupportsScanUnsafeRow extends DataSourceV2Reader {
public interface SupportsScanUnsafeRow extends DataSourceReader {

@Override
default List<DataReaderFactory<Row>> createDataReaderFactories() {
Expand All @@ -39,7 +39,7 @@ default List<DataReaderFactory<Row>> createDataReaderFactories() {
}

/**
* Similar to {@link DataSourceV2Reader#createDataReaderFactories()},
* Similar to {@link DataSourceReader#createDataReaderFactories()},
* but returns data in unsafe row format.
*/
List<DataReaderFactory<UnsafeRow>> createUnsafeRowReaderFactories();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.apache.spark.sql.sources.v2.DataSourceV2;
import org.apache.spark.sql.sources.v2.DataSourceV2Options;
import org.apache.spark.sql.sources.v2.streaming.writer.StreamWriter;
import org.apache.spark.sql.sources.v2.writer.DataSourceV2Writer;
import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.streaming.OutputMode;
import org.apache.spark.sql.types.StructType;

Expand All @@ -31,15 +31,15 @@
* provide data writing ability for structured streaming.
*/
@InterfaceStability.Evolving
public interface StreamWriteSupport extends BaseStreamingSink {
public interface StreamWriteSupport extends DataSourceV2, BaseStreamingSink {

/**
* Creates an optional {@link StreamWriter} to save the data to this data source. Data
* sources can return None if there is no writing needed to be done.
*
* @param queryId A unique string for the writing query. It's possible that there are many
* writing queries running at the same time, and the returned
* {@link DataSourceV2Writer} can use this id to distinguish itself from others.
* {@link DataSourceWriter} can use this id to distinguish itself from others.
* @param schema the schema of the data to be written.
* @param mode the output mode which determines what successive epoch output means to this
* sink, please refer to {@link OutputMode} for more details.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.execution.streaming.BaseStreamingSource;
import org.apache.spark.sql.sources.v2.reader.DataSourceV2Reader;
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;

import java.util.Optional;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to allow reading in a continuous processing mode stream.
*
* Implementations must ensure each reader factory output is a {@link ContinuousDataReader}.
Expand All @@ -33,7 +33,7 @@
* DataSource V1 APIs. This extension will be removed once we get rid of V1 completely.
*/
@InterfaceStability.Evolving
public interface ContinuousReader extends BaseStreamingSource, DataSourceV2Reader {
public interface ContinuousReader extends BaseStreamingSource, DataSourceReader {
/**
* Merge partitioned offsets coming from {@link ContinuousDataReader} instances for each
* partition to a single global offset.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,20 @@
package org.apache.spark.sql.sources.v2.streaming.reader;

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.sources.v2.reader.DataSourceV2Reader;
import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
import org.apache.spark.sql.execution.streaming.BaseStreamingSource;

import java.util.Optional;

/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* A mix-in interface for {@link DataSourceReader}. Data source readers can implement this
* interface to indicate they allow micro-batch streaming reads.
*
* Note: This class currently extends {@link BaseStreamingSource} to maintain compatibility with
* DataSource V1 APIs. This extension will be removed once we get rid of V1 completely.
*/
@InterfaceStability.Evolving
public interface MicroBatchReader extends DataSourceV2Reader, BaseStreamingSource {
public interface MicroBatchReader extends DataSourceReader, BaseStreamingSource {
/**
* Set the desired offset range for reader factories created from this reader. Reader factories
* will generate only data within (`start`, `end`]; that is, from the first record after `start`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,19 @@
package org.apache.spark.sql.sources.v2.streaming.writer;

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.sources.v2.writer.DataSourceV2Writer;
import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.sources.v2.writer.DataWriter;
import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;

/**
* A {@link DataSourceV2Writer} for use with structured streaming. This writer handles commits and
* A {@link DataSourceWriter} for use with structured streaming. This writer handles commits and
* aborts relative to an epoch ID determined by the execution engine.
*
* {@link DataWriter} implementations generated by a StreamWriter may be reused for multiple epochs,
* and so must reset any internal state after a successful commit.
*/
@InterfaceStability.Evolving
public interface StreamWriter extends DataSourceV2Writer {
public interface StreamWriter extends DataSourceWriter {
/**
* Commits this writing job for the specified epoch with a list of commit messages. The commit
* messages are collected from successful data writers and are produced by
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
* Please refer to the documentation of commit/abort methods for detailed specifications.
*/
@InterfaceStability.Evolving
public interface DataSourceV2Writer {
public interface DataSourceWriter {

/**
* Creates a writer factory which will be serialized and sent to executors.
Expand Down
Loading