[FEATURE] Add score attribute for suggestions associated to a multi label question

[jfcalvo]

For the new feature adding support for span questions we will give support to specify a score for suggestions associated to a span question.

For that we are following the next format (PUT to upsert a suggestion):

{
    "question_id": "id-of-the-question",
    "type": "model",
    "value": [
        {"label": "label-a", "start": 0, "end": 1, "score": 0.2},
        {"label": "label-b", "start": 2, "end": 3, "score": 1},
        {"label": "label-c", "start": 4, "end": 5},
    ]
}

For this issue we should add score attribute for suggestions associated to a multi_label_selection question too. With this change users can specify a score value for every label inside the suggestion.

Additional points to research/discuss:

• Changes to the SDK.
• Changes related to UI. Are we gonna show this information in the UI?

Tasks

Preview Give feedback

1. [TASK] Allow score suggestion attribute to support a list of values argilla-server#93
   +0
   
2. [TASK] Create a rollback process for version downgrades argilla-server#113
   +0
   
3. [TASK] Improve score migration to JSON so scores associated to suggestion with multi values are set to NULL argilla-server#128
   +0
   
4. [TASK] Remove score attribute support for span question suggestions argilla-server#99
   +0
   
5. [TASK] Add changes on SDK to support suggestion score attribute accepting list of values #4724
   area: python sdk language: python severity: minor team: backend type: enhancement +5
   
6. Review and update search index mapping to support the change #4725
   area: server severity: minor team: backend type: enhancement +4
   

[dosubot]

To accommodate the proposed feature of adding a score attribute for suggestions associated with a multi_label_selection question in the Argilla repository, modifications would be necessary in both the SDK and the UI. Here's a breakdown of the changes required:

SDK Modifications

1. Extend the MultiLabelQuestion Schema:
   • You need to add a new attribute to the MultiLabelQuestion class to hold the score for each label. This could be a dictionary mapping labels to their scores.

class MultiLabelQuestion(_LabelQuestion):
    # Existing attributes...
    scores: Optional[Dict[str, float]] = None

    @property
    def server_settings(self) -> Dict[str, Any]:
        settings = super().server_settings
        if self.scores:
            settings["scores"] = self.scores
        return settings

2. Update the server_settings Property:
   • Modify the server_settings property of the MultiLabelQuestion class to include the scores in the payload if they are provided.

UI Modifications

1. Display Scores in the UI:

   • The UI component responsible for rendering multi_label_selection questions needs to be updated to display the score next to each label if scores are provided. This might involve adjusting the layout to accommodate the display of scores in a user-friendly manner.

2. Update Data Handling:

   • Ensure that the UI component handling the submission of answers for multi_label_selection questions can correctly handle and submit the scores along with the selected labels if required.

3. Validation and Error Handling:

   • Implement validation to ensure that scores, if used, are within an acceptable range (e.g., 0 to 1 or 1 to 100, depending on the scoring system you choose). Also, add error handling to provide feedback to the user if the scores are not in the expected format or range.

Considerations

• Backward Compatibility: Ensure that the addition of the scores attribute does not break existing functionality for multi_label_selection questions that do not use this new feature.
• Documentation: Update the SDK documentation and UI component documentation to reflect the new feature and provide examples of how to use it.
• Testing: Add unit tests and UI tests to cover the new functionality and ensure that it works as expected across different scenarios.

These modifications will enable the Argilla SDK and UI to support the new feature of adding scores to suggestions for multi_label_selection questions, enhancing the flexibility and utility of the feedback collection process.

For reference, the current structure of the MultiLabelQuestion class in the Argilla SDK is as follows:

class MultiLabelQuestion(_LabelQuestion):
    """Schema for the `FeedbackDataset` label questions, which are the ones that will
    require a label response from the user. This class should be used when the user can
    select multiple labels.

    Args:
        type: The type of the question. Defaults to 'multi_label_selection' and
            cannot/shouldn't be modified.
        labels: The list of labels of the label question. The labels must be unique, and
            the list must contain at least two unique labels. Additionally, `labels` can
            also be a dictionary of labels, where the keys are the labels, and the values
            are the labels that will be shown in the UI.
        visible_labels: The number of visible labels in the UI. Defaults to 20, and must
            be 3 or greater.

    Examples:
        >>> from argilla.client.feedback.schemas.questions import MultiLabelQuestion
        >>> MultiLabelQuestion(name="multi_label_question", title="Multi Label Question", labels=["label_1", "label_2"])
    """

    type: Literal[QuestionTypes.multi_label_selection] = Field(
        QuestionTypes.multi_label_selection.value, allow_mutation=False
    )

You can find the source code for this class in the Argilla repository.

Sources

• src/argilla/client/feedback/schemas/questions.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.