Fix msg "'<' not supported between instances of 'str' and 'int'" #4236
Conversation
Druid returns NULL as 0, typed as int. This causes pandas to fail when it tries to sort heterogeneous types.
mistercrunch
force-pushed
the
fix_pandas_issue
branch
from
631135e to
1e4f6d8
Compare
betodealmeida
left a comment
betodealmeida
left a comment
There was a problem hiding this comment.
Should we change pydruid to do this automatically, or maybe convert the zero to None?
| cols += query_obj.get('groupby') or [] | ||
| cols += query_obj.get('columns') or [] | ||
| cols += query_obj.get('metrics') or [] | ||
| cols += groupby |
There was a problem hiding this comment.
cols = [DTTM_ALIAS] + groupby + columns + metrics
may be a bit cheaper and cleaner
| cols = [col for col in cols if col in df.columns] | ||
| df = df[cols] | ||
|
|
||
| for col in groupby + columns: |
There was a problem hiding this comment.
would it be possible for some of these columns to not be in df.columns given the list comprehension at line 1249?
|
@betodealmeida I agree that pydruid should take care of that, but that change may not be backward compatible, so I'm not sure how to handle it. I have to admit I'm confused to see the related bugs. I feel like it's a new behavior (new version of Druid? PyDruid? error in druid ingestion?) as I we should have had this problem before... related? |
|
Dug deeper and got to the root cause in #4358 |
3 participants
Druid returns NULL as 0, typed as int. This causes pandas to fail
when it tries to sort heterogeneous types.