reset_index is super slow

[luistelmocosta]

Hello, I am trying to load a dataset that does not have a particular index and I would like to know if my approach is correct.

I read that due to koalas index issues we should always specify the index, so I did it like this:

df = df_tmp.to_koalas(index_col = ['A', 'B']) since A and B are the columns that make a value unique.

However I need to drop the index because I will need these columns further and I tried to reset_index():

df = df.sort_values('Date').reset_index()

But this opperation takes 32 minutes which is completely infeasible.

My dataset comes from a csv file

df_tmp = spark.read.option("header", "true").csv(data_lake_path+"test.csv")

And I added this line already:

ks.set_option('compute.default_index_type', 'distributed-sequence')

Any idea what I am doing wrong?

[HyukjinKwon]

I think the root cause is that we don't support the assignment of index to the columns. I will make a quick fix.

>>> kdf = koalas.range(1000)
>>> kdf["col"] = kdf.index

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../koalas/databricks/koalas/frame.py", line 10075, in __setitem__
    kdf = self._assign({key: value})
  File "/.../koalas/databricks/koalas/frame.py", line 4301, in _assign
    "Column assignment doesn't support type " "{0}".format(type(v).__name__)
TypeError: Column assignment doesn't support type Index

[ueshin]

I'm fine with the fix at #1696, but I'm not sure it will solve the issue here.

If the index after reset_index() needs to be sequential values, it won't fix.
Attaching distributed-sequence default index could still be heavy. Cache could be helpful:

df = df.sort_values('Date').spark.cache().reset_index()

If the index after reset_index() doesn't care the values, you can just use distributed default index when resetting.

with ks.option_context('compute.default_index_type', 'distributed'):
    df = df.reset_index()

Btw, if it's okay to leave the indices as they are, you can also use reset_index(drop=False) which won't cause attaching the default index. nvm, I misunderstood the usage of it.