Skip to content

[SPARK-22452][SQL]Add getInt, getLong, getBoolean to DataSourceV2Options #19902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
skambha wants to merge 5 commits into from
Closed

[SPARK-22452][SQL]Add getInt, getLong, getBoolean to DataSourceV2Options #19902

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
566b7a5
Add getInt, getLong, getBoolean to DataSourceV2Options and add unit t…
skambha Dec 5, 2017
bb01ff5
Fix formatting
skambha Dec 5, 2017
15e8ce5
Add methods that take defaultValue and return the primitive
skambha Dec 6, 2017
5de7962
Remove the version of the 3 methods that doesnt take a default value
skambha Dec 6, 2017
7d87edc
Add comments
skambha Dec 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2Options.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,22 @@ public DataSourceV2Options(Map<String, String> originalMap) {
public Optional<String> get(String key) {
return Optional.ofNullable(keyLowerCasedMap.get(toLowerCase(key)));
}

@gatorsmile gatorsmile Dec 6, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/** Get a parameter as a boolean, falling back to a default if not set */

@skambha skambha Dec 6, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Is it ok if I modify the comment for the method a bit to this:

  /**
   * Returns the boolean value to which the specified key is mapped, 
   * or defaultValue if there is no mapping for the key.
   * The key match is case-insensitive
   */

public Optional<Boolean> getBoolean(String key) {

@cloud-fan cloud-fan Dec 6, 2017 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to avoid the expensive boxing, how about public boolean getBoolean(String key, boolean defaultValue)? This is also consistent with SparkConf.getXXX

@skambha skambha Dec 6, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea.

  1. So I have implemented the getBoolean, getInt, and getLong using the defaultValue and added unit tests for it.

  2. For now, I have left the other methods as well that return Optional<> keeping in line with the getString as it may still have some value.

Please take a look.

If we don't want (2), I can remove the version of the getBoolean, getLong, getInt that doesn't take the defaultValue.

Thanks for your comments.

@skambha skambha Dec 6, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per @gatorsmile , I have gone and removed the version of the methods that doesn't take the default value.

String lcaseKey = toLowerCase(key);
return Optional.ofNullable(keyLowerCasedMap.containsKey(lcaseKey) ?
Boolean.parseBoolean(keyLowerCasedMap.get(lcaseKey)) : null);
}

public Optional<Integer> getInt(String key) {
String lcaseKey = toLowerCase(key);
return Optional.ofNullable(keyLowerCasedMap.containsKey(lcaseKey) ?
Integer.parseInt(keyLowerCasedMap.get(lcaseKey)) : null);
}

public Optional<Long> getLong(String key) {

@gatorsmile gatorsmile Dec 6, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also remove the three APIs without the default values?

@skambha skambha Dec 6, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Let me post another commit. Thanks.

String lcaseKey = toLowerCase(key);
return Optional.ofNullable(keyLowerCasedMap.containsKey(lcaseKey) ?
Long.parseLong(keyLowerCasedMap.get(lcaseKey)) : null);
}
}
29 changes: 29 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2OptionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,33 @@ class DataSourceV2OptionsSuite extends SparkFunSuite {
val options = new DataSourceV2Options(Map("foo" -> "bAr").asJava)
assert(options.get("foo").get == "bAr")
}

test("getInt") {
val options = new DataSourceV2Options(Map("numFoo" -> "1", "foo" -> "bar").asJava)
assert(options.getInt("numFoo").get == 1)

intercept[NumberFormatException]{
options.getInt("foo").isPresent
}
}

test("getBoolean") {
val options = new DataSourceV2Options(
Map("isFoo" -> "true", "isFoo2" -> "false", "foo" -> "bar").asJava)
assert(options.getBoolean("isFoo").get == true)
assert(options.getBoolean("isFoo2").get == false)
assert(options.getBoolean("isFoo2").isPresent)
assert(options.getBoolean("foo").get == false)
assert(!options.getBoolean("isBar").isPresent)
}

test("getLong") {
val options = new DataSourceV2Options(Map("numFoo" -> "9223372036854775807",
"foo" -> "bar").asJava)
assert(options.getLong("numFoo").get == 9223372036854775807L)

intercept[NumberFormatException]{
options.getLong("foo").isPresent
}
}
}