Fix fasttext predict call for numpy>2

[ayushdg]

Description

Fixes Fasttext stages failing with numpy>=2.0.0

The solution is to pass in text as a list as discussed here thanks @sarahyurick for suggesting.

The results are one level more nested and now return a np.float32 scalar so added a another level of indexing and .item call to get the raw float.

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Overview

Greptile Summary

Fixed fasttext model predict calls to be compatible with numpy>=2.0.0 by wrapping text input in a list and adjusting the result indexing.

Key Changes:

• Changed model.predict(text) to model.predict([text]) in both FastTextQualityFilter and FastTextLangId
• Updated indexing from score[0] to score[0][0].item() to handle the extra nesting level and convert np.float32 scalar to Python float
• Updated label indexing from label[0] to label[0][0] to match the new result structure

The fix aligns with the recommended solution from the Hugging Face community for handling the API change in fasttext with numpy 2.x.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk
• The changes are minimal, well-targeted, and address a specific compatibility issue with numpy>=2.0.0. The fix follows the recommended solution from the fasttext community and maintains backward compatibility in behavior while adapting to the new API structure. Both affected methods (FastTextQualityFilter.score_document and FastTextLangId.score_document) receive identical treatment, ensuring consistency.
• No files require special attention

Important Files Changed

Filename	Overview
nemo_curator/stages/text/filters/fasttext_filter.py	Updated fasttext predict calls to use list input format for numpy>=2.0.0 compatibility, added extra indexing and .item() calls

[greptile-apps]

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[sarahyurick]

I tested both the quality and language models and both worked for me. Here is how I tested the quality model:

from nemo_curator.core.client import RayClient
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters.fasttext_filter import FastTextQualityFilter
from nemo_curator.stages.text.io.reader import ParquetReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.modules import ScoreFilter


client = RayClient(num_cpus=16, num_gpus=0)
client.start()

pipeline = Pipeline("fasttext_filter_test")
pipeline.add_stage(ParquetReader("..."))
pipeline.add_stage(ScoreFilter(FastTextQualityFilter("model_quantized.bin", label="__label__High"), score_field="quality"))
pipeline.add_stage(JsonlWriter("./quality_results"))
pipeline.run()

client.stop()

where I downloaded a model here: https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2 and its outputs are __label__High, __label__Mid, and __label__Low.