Add an option to enable operations on different DataFrames

[HyukjinKwon]

This PR proposes to add an environment variable, OPS_ON_DIFF_FRAMES, to enable operations on different DataFrames.

To use this feature, set OPS_ON_DIFF_FRAMES environment variable to true and run Koalas codes.

The changes here are a bit big to match the behaviours with pandas, and to generalize them. However, how it works is pretty straightforward. Basically, it joins in index columns and do the operations. See the rough Spark examples below:

>>> df1.show()

+-----------------+---+
|__index_level_0__|  a|
+-----------------+---+
|                0|  1|
|                1|  2|
|                2|  3|
+-----------------+---+

>>> df2.show()

+-----------------+---+
|__index_level_0__|  a|
+-----------------+---+
|                0|  1|
|                1|  2|
|                2|  3|
+-----------------+---+

>>> df1.join(df2, on="__index_level_0__", how="full").show()

+-----------------+---+---+
|__index_level_0__|  a|  a|
+-----------------+---+---+
|                0|  1|  1|
|                1|  2|  2|
|                2|  3|  3|
+-----------------+---+---+

Annoying part here is that how to resolve duplicated column names a. In the current implementation, it's kind of tricky to use DataFrame's alias.

In this PR, I had to workaround by aliasing it. For instance:

+-----------------+--------+--------+
|__index_level_0__|__this_a|__that_a|
+-----------------+--------+--------+
|                0|       1|       1|
|                1|       2|       2|
|                2|       3|       3|
+-----------------+--------+--------+

and then, perform the operations, e.g.:

+-----------------+---------------------+
|__index_level_0__|(__this_a + __that_a)|
+-----------------+---------------------+
|                0|                    2|
|                1|                    4|
|                2|                    6|
+-----------------+---------------------+

and alias back:

+-----------------+---+
|__index_level_0__|  a|
+-----------------+---+
|                0|  2|
|                1|  4|
|                2|  6|
+-----------------+---+

Resolves #624

[HyukjinKwon]

There's some docs at align_diff_frames about how the function should be given to align_diff_frames.

[codecov-io]

Codecov Report

Merging #633 into master will increase coverage by 0.07%.
The diff coverage is 95.77%.

@@            Coverage Diff             @@
##           master     #633      +/-   ##
==========================================
+ Coverage   92.95%   93.02%   +0.07%     
==========================================
  Files          31       31              
  Lines        5093     5222     +129     
==========================================
+ Hits         4734     4858     +124     
- Misses        359      364       +5

Impacted Files	Coverage Δ
databricks/koalas/base.py	92.07% <100%> (+0.24%)	⬆️
databricks/koalas/frame.py	94.57% <89.65%> (-0.07%)	⬇️
databricks/koalas/utils.py	97.9% <97.11%> (-2.1%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82e2e41...18e883c. Read the comment docs.

[softagram-bot]

Softagram Impact Report for pull/633 (head commit: 18e883c)

⭐ Change Overview

(Open in Softagram Desktop for full details)

⭐ Details of Dependency Changes

(Open in Softagram Desktop for full details)

📄 Full report

• Permalink: Full report for pull/633

Give feedback on this report to [email protected]

[HyukjinKwon]

I am merging this to unblock #624