Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DataFrame.spark.hint to reflect internal changes. #1865

Merged
merged 1 commit into from
Oct 24, 2020

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Oct 23, 2020

Fixes DataFrame.spark.hint to reflect internal changes.

E.g.,:

>>> kdf1 = ks.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'], 'value': [1, 2, 3, 5]}).set_index('lkey')
>>> kdf2 = ks.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'], 'value': [5, 6, 7, 8]}).set_index('rkey')
>>> kdf1.merge((kdf2 + 1).spark.hint("broadcast"), left_index=True, right_index=True).sort_values(['value_x', 'value_y'])
      value_x  value_y
lkey
foo         1        5
foo         1        8
bar         2        6
baz         3        7
foo         5        5
foo         5        8

This is different from the result without the hint:

>>> kdf1.merge(kdf2 + 1, left_index=True, right_index=True).sort_values(['value_x', 'value_y'])
      value_x  value_y
lkey
foo         1        6
foo         1        9
bar         2        7
baz         3        8
foo         5        6
foo         5        9

@codecov-io
Copy link

codecov-io commented Oct 23, 2020

Codecov Report

Merging #1865 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1865   +/-   ##
=======================================
  Coverage   94.14%   94.14%           
=======================================
  Files          40       40           
  Lines        9792     9793    +1     
=======================================
+ Hits         9219     9220    +1     
  Misses        573      573           
Impacted Files Coverage Δ
databricks/koalas/internal.py 96.44% <100.00%> (ø)
databricks/koalas/spark/accessors.py 93.07% <100.00%> (+0.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d0b2f2...9da874d. Read the comment docs.

@HyukjinKwon HyukjinKwon merged commit deefc2c into databricks:master Oct 24, 2020
@ueshin ueshin deleted the fix_hint branch October 28, 2020 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants