Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Content Type Classifier #361

Merged
merged 15 commits into from
Dec 16, 2024
Merged

Content Type Classifier #361

merged 15 commits into from
Dec 16, 2024

Conversation

sarahyurick
Copy link
Collaborator

@sarahyurick sarahyurick commented Nov 13, 2024 •
edited
Loading

Awaiting Hugging Face release: https://huggingface.co/nvidia/content-type-classifier-deberta.

labels = [
    "Product/Company/Organization/Personal Websites",
    "Explanatory Articles",
    "News",
    "Blogs",
    "MISC",
    "Boilerplate Content",
    "Analytical Exposition",
    "Online Comments",
    "Reviews",
    "Books and Literature",
    "Conversational",
]

@sarahyurick
add initial changes
3592996 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick
add python scripts
461cc2e 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick
edit docs
b198d3e 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick
update max_len
5ec37eb 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick sarahyurick mentioned this pull request Nov 21, 2024
Closed
@sarahyurick sarahyurick mentioned this pull request Dec 6, 2024
Open
7 tasks
@sarahyurick sarahyurick changed the title Type of Speech Classifier Content Type Classifier Dec 9, 2024
@sarahyurick
edit docstring
6d915ea 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick sarahyurick marked this pull request as ready for review December 9, 2024 20:56
@sarahyurick sarahyurick requested a review from VibhuJawa December 9, 2024 21:05
VibhuJawa
VibhuJawa requested changes Dec 16, 2024
View reviewed changes
Copy link
Collaborator

@VibhuJawa VibhuJawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, minor feedback

nemo_curator/classifiers/content_type.py Outdated Show resolved Hide resolved
nemo_curator/scripts/classifiers/README.md Outdated Show resolved Hide resolved
nemo_curator/classifiers/content_type.py Outdated Show resolved Hide resolved
sarahyurick and others added 2 commits December 16, 2024 10:41
VibhuJawa
VibhuJawa approved these changes Dec 16, 2024
View reviewed changes
Copy link
Collaborator

@VibhuJawa VibhuJawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

nemo_curator/classifiers/content_type.py Outdated
ContentTypeClassifier is a text classification model designed to categorize documents into one of 11 distinct speech types based on their content.
It analyzes and understands the nuances of textual information, enabling accurate classification across a diverse range of content types.
The pretrained model used by this class can be found on Hugging Face here: https://huggingface.co/nvidia/content-type-classifier-deberta.
This class is optimized for running on multi-node, multi-GPU setups to enable fast and efficient inference on large datasets.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This class is optimized for running on multi-node, multi-GPU setups to enable fast and efficient inference on large datasets.
This classifier is optimized for running on multi-node, multi-GPU setups to enable fast and efficient inference on large datasets.

@sarahyurick
edit docstrings
80e9095 
Signed-off-by: Sarah Yurick <[email protected]>
@sarahyurick sarahyurick merged commit 9df5d7b into NVIDIA:main Dec 16, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VibhuJawa VibhuJawa VibhuJawa approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sarahyurick @VibhuJawa