Error in filtering

[dharik-arsath]

I have some empty strings and so I used df = df[~(df.x == ' ')]. Now if I check that column things are gone and the data is clean but the column in the string datatype so if I changed that info float it and things are alright here. but when I do some filtering on that column it is throwing me the error that column has empty string. Though I have cleaned and reassigned with df.

I can able to plot histogram, I can able to see even minmax, but when I filter that column it is throwing me the error.What is the problem here? I have tried to materialize as well that also throws me the same error. Please look into the issue. Thank you

When I check for df.x it is giving me the expression with float 32 which is as expected but why can't I filter it like df[df.x > 50000]

[JovanVeljanoski]

Can you post a reproducible example?

[JovanVeljanoski]

By reproducible example I mean a small code snippet that reproduces the issue. Something we can copy-paste in our local environment as see what happens. Like a unit-test basically. Thank would help. Feel free to submit a PR with a unit-test also if you are up for it.

[JovanVeljanoski]

Ah ok yeah something like that, but including a small aprt of the data. I can't run your example as it is.

[JovanVeljanoski]

Please just do a minimal reproducible example.

[JovanVeljanoski]

Ok you can track the progress of this in:
#1421

For future reference, this is the kind of code I was looking for:

import vaex
df = vaex.from_dict({'x': ['1', '2.3', '', '', '5', '-5']})
df = df[~(df.x == '')]
df['x'] = df.x.astype('float')
df = df[df.x > 0]

If you provide things like this in the future, we can get to your problem faster, and be more motivated to fix things generally. Thanks.

[dharik-arsath]

import vaex
my_data = {'TotalCharges':[1000,2000,' ',' ',500]}
data = vaex.from_dict(my_data)
data = data[~(data.TotalCharges == ' ')]
data['TotalCharges'] = data.TotalCharges.astype('int16')
data[data.TotalCharges > 500]

infact data.TotalCharges > 500 gives boolean indexing perfectly when I convert into dataframe it shows the error

I hope I can able to produce code as you want. Thanks