[docs] update load_tool

[faaany]

What does this PR do?

This model m-ric/text-to-image seems to not exit anymore on the HuggingFace Model Hub. When running the example code, I got the following error:

You're loading a tool from the Hub from None. Please make sure this is a source that you trust as the code within that tool will be executed on your machine. Always verify the code of the tools that you load. We recommend specifying a `revision` to ensure you're loading the code that you have checked.
Traceback (most recent call last):
  File "/home/sdp/fanli/transformers/training.py", line 10, in <module>
    image_generation_tool = load_tool("m-ric/text-to-image")
  File "/home/sdp/fanli/transformers/src/transformers/agents/tools.py", line 801, in load_tool
    return Tool.from_hub(task_or_repo_id, model_repo_id=model_repo_id, token=token, **kwargs)
  File "/home/sdp/fanli/transformers/src/transformers/agents/tools.py", line 357, in from_hub
    return tool_class(**kwargs)
  File "/home/sdp/fanli/transformers/src/transformers/agents/tools.py", line 97, in new_init
    self.validate_arguments(do_validate_forward=do_validate_forward)
  File "/home/sdp/fanli/transformers/src/transformers/agents/tools.py", line 152, in validate_arguments
    raise TypeError(f"You must set an attribute {attr} of type {expected_type.__name__}.")
TypeError: You must set an attribute inputs of type Dict.

I only found this page, so I used the code from there to update the doc. If it is better to use load tool directly from model hub, pls advice me one, so I can update my PR to use a working image_generation_tool.

@stevhliu

[stevhliu]

Good catch, thanks! Can you maybe upload the tool to the Hub so we can do a more direct replacement of the existing code snippet?

[sergiopaniego]

Hi! 👋

I run into the same problem since it is also happening in the cookbooks.
This code (the cookbook) was previously working and the tool is actually available and wasn't removed 😄
I believe that the problem comes from the type check done in tools.py:

required_attributes = {
    "description": str,
    "name": str,
    "inputs": Dict,
    "output_type": str,
}
authorized_types = ["string", "integer", "number", "image", "audio", "any", "boolean"]

for attr, expected_type in required_attributes.items():
    attr_value = getattr(self, attr, None)
    if not isinstance(attr_value, expected_type):
        raise TypeError(f"You must set an attribute {attr} of type {expected_type.__name__}.")

As stated, it expects a Dict for the inputs instead of a dict.

A possible solution would be updating the m-ric/text-to-image using Dict. Another possible solution would be changing the validate_arguments to be more flexible.

I could implement any of them if help is needed 😄

[sergiopaniego]

Experimenting further, I've found that when loading a tool from the hub, the method from_hub is called and in this line:

transformers/src/transformers/agents/tools.py

Line 352 in 25f510a

if tool_class.inputs != custom_tool["inputs"]:

The inner code is executed:

transformers/src/transformers/agents/tools.py

Line 353 in 25f510a

tool_class.inputs = custom_tool["inputs"]

The first value tool_class.inputs is a dict, which is correct whereas the second value custom_tool["inputs"] is an string. Essentially, this causes the issue.

# tool_class.inputs
<class 'dict'>
{'prompt': {'type': 'string', 'description': "The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc."}}

# custom_tool["inputs"]
<class 'str'>
{'prompt': {'type': 'text', 'description': "The image generator prompt. Don't hesitate to add details in the prompt to make the image look better, like 'high-res, photorealistic', etc."}}

Commenting out these two lines, the code runs correctly without any issue.

cc @aymeric-roucher 😄

[sergiopaniego]

What do you think @stevhliu?

[sergiopaniego]

For the other example in Agents, supercharged - Multi-agents, External tools, and more model_download_tool = load_tool("m-ric/hf-model-downloads"). The result is similar in the same place:

# tool_class.inputs
<class 'dict'>
{'task': {'type': <class 'str'>, 'description': 'the task category (such as text-classification, depth-estimation, etc)'}}

# custom_tool["inputs"]
<class 'str'>
{'task': {'type': <class 'str'>, 'description': 'the task category (such as text-classification, depth-estimation, etc)'}}

In order to make it work, in addition to commenting out the previously mentioned lines, you must also add in that same place the following:

tool_class.output_type = 'string'

I think that the issue comes from:

• custom_tool["inputs"] which is not correctly converted to a dict
• The custom Tool definition may have changed a bit since HFModelDownloadsTool and TextToImageTool differ a little bit.

I'm happy to review it further and generate a PR if needed 😄

[stevhliu]

Wow thanks so much for your investigation @sergiopaniego! I'm in favor of commenting out those two lines, but let's wait and see what @aymeric-roucher thinks 🙂

[aymeric-roucher]

Well spotted @faaany and @sergiopaniego ! This is indeed due to the assignment of tool_class.inputs.

I've made a fix for this input type error just a few hours ago while adding new stuff in this PR (merged now). So your issue should be fixed now!

And I've manually fixed the tools, all should work now. The rendering for https://huggingface.co/spaces/m-ric/hf-model-downloads still fails in the Space (you can still pull the tool and it works properly) but will be merged in a PR very soon!

[faaany]

Hi all, thanks so much for the discussion! Closing this PR, as it got fixed in @aymeric-roucher 's PR.