[FEATURE] Make the limit in the number of labels for multi-label classification tasks configurable (possibly higher than 250)

[davidefiocco]

At the moment, when dealing with (multi-label) classification tasks with very many labels num_labels (>250), as in

import argilla as rg
api_url = "http://localhost:6900"
api_key = "<api-key>"

rg.init(api_url=api_url, api_key=api_key)

#rg.Workspace.create('argilla')

rg.set_workspace("argilla")

num_labels = 251

labels = ["label " + str(i) for i in range(0, num_labels)]

dataset = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="text", required= True, use_markdown=True),
    ],
    questions=[
        rg.MultiLabelQuestion(name="question-multi", title="Which labels are correct?", labels=labels, required=True),
    ]
)

records = []

record = rg.FeedbackRecord(
    fields={
        "text": "tierra finamente dividida, constituida por agregados de silicatos de aluminio hidratados"
    },
    suggestions=[{"question_name": "question-multi", 
                  "value":["label 28", "label 8", "label 53", "label 14", "label 5"], 
                  "agent": "my-first-model"}]
)
records.append(record)

dataset.add_records(records)

dataset.push_to_argilla(name="way-too-many-labels-dataset", workspace="argilla")

dataset creation fails with a

Exception: Failed while adding the question 'question-multi' to the `FeedbackDataset` in Argilla with exception: Argilla server returned an error with http status: 422. Error details: 
{'code': 'argilla.api.errors::ValidationError', 'params': {'errors': [{'loc': ['body', 'settings', 'MultiLabelSelectionQuestionSettingsCreate', 'options'], 'msg': 'ensure this value has 
at most 250 items', 'type': 'value_error.list.max_items', 'ctx': {'limit_value': 250}, 'value': {}}]}}

I believe that this many-label scenario can appear e.g. in retail/product classification annotation workloads.

Ideally, I would like the limit_value to be configurable when running the argilla server, e.g. as an env var.

A current workaround is to subset datasets into chunks whose examples are likely to contain only a limited <250 subset of labels. This is possible e.g. by exploiting the taxonomy of labels, but can be clunky and relies on the assumptions that none of the excluded labels appears in the various subsets.

Additional context
I am using argilla=1.24.0 and the quickstart server pulled via docker pull argilla/argilla-quickstart:latest.
This was discussed with @jfcalvo and @davidberenstein1957 over Slack on the 02-support-and-questions channel. Thank you so much!

[dosubot]

🤖

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[davidefiocco]

For what it's worth, I tried testing a workaround editing in the quickstart docker container the relevant variable

https://github.com/argilla-io/argilla-server/blob/efe3bceaeeeff8d8738ed7afc271750994c94767/src/argilla_server/schemas/v1/questions.py#L49

To do so, within a root shell terminal in docker I increased by 10x the limit in the number of labels:

sed -i 's/LABEL_SELECTION_OPTIONS_MAX_ITEMS = 250/LABEL_SELECTION_OPTIONS_MAX_ITEMS = 2500/' ./opt/venv/lib/python3.10/site-packages/argilla_server/schemas/v1/questions.py
sed -i 's/LABEL_SELECTION_OPTIONS_MAX_ITEMS = 250/LABEL_SELECTION_OPTIONS_MAX_ITEMS = 2500/' ./opt/venv/lib/python3.10/site-packages/argilla/server/schemas/v1/questions.py

restarted the container, and executed the code in #4615 (comment) setting num_labels = 1500. The error I mentioned goes away.

It's not a proper test (I haven't started annotations), but the frontend seems to work fine (it's responsive) with that number of labels:

[davidefiocco]

It's anecdotal, but when I experiment on a remote Argilla server, clicking on "Less" to fold/unfold the long list of labels, makes the UI fail to respond, and one may need to refresh the page.

Things work well when running locally on Docker instead.

So this issue may need more work than a variable change to work well in the frontend :(

[davidberenstein1957]

@davidefiocco , sad to hear that it is not plug-and-play but thanks a lot for the additional work on the topic :)

[jfcalvo]

@davidefiocco with the coming release v1.27.0 we added new environment variables so now it's possible to change the limits (at instance level) for label/multi-label and span questions.

These changes will be added to the documentation once the release is published. In case you want to take a look to the docs changes previous the publishing you can see them here: https://github.com/argilla-io/argilla/pull/4713/files#diff-7985416b05c65eb8bdc26219fec4d817b1e9bb725a599d9e88841c64b1539a40R88