Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update make_data_column_uniform #125

Open
wants to merge 4 commits into
base: new-aliasing-structure
Choose a base branch
from
Open

update make_data_column_uniform #125

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6c4f601
update make_data_column_uniform
AmeyHengle Jan 29, 2023
98ccd74
Update preprocessing.py
AmeyHengle Mar 1, 2023
1215b82
Update preprocessing.py
AmeyHengle Mar 1, 2023
0b4e5e9
Update preprocessing.py
AmeyHengle Mar 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions slu/slu/utils/preprocessing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def make_reftime_column_uniform(data_frame: pd.DataFrame) -> pd.DataFrame:

return data_frame

def make_data_column_uniform(data_frame: pd.DataFrame) -> pd.DataFrame:
def make_data_column_uniform(data_frame: pd.DataFrame) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does the type hint need a change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a mistake, return type hint shouldn't be removed. Fixed this in a new commit.

if const.ALTERNATIVES in data_frame.columns:
column = const.ALTERNATIVES
elif const.DATA in data_frame.columns:
Expand All @@ -74,9 +74,11 @@ def make_data_column_uniform(data_frame: pd.DataFrame) -> pd.DataFrame:
):
if isinstance(row[const.ALTERNATIVES], str):
data = json.loads(row[const.ALTERNATIVES])
if isinstance(data, str):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would json.loads yield a string? Shouldn't it strictly give a dict?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Production calls in vodafone contain noisy transcripts, where ASR alternatives seem to be dumped twice. The second if condition is supposed to fix this by performing another json.loads(data).

Case example:
image

data = json.loads(row[const.ALTERNATIVES])
if const.ALTERNATIVES in data:
data_frame.loc[i, const.ALTERNATIVES] = json.dumps(
data[const.ALTERNATIVES]
data[const.ALTERNATIVES], ensure_ascii=False
)

return data_frame