Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GGUF as supported expand[] parameter #2545

Merged
merged 2 commits into from
Sep 16, 2024
Merged

Conversation

Wauplin
Copy link
Contributor

@Wauplin Wauplin commented Sep 16, 2024

Follow-up after https://github.com/huggingface-internal/moon-landing/pull/11060 and https://github.com/huggingface-internal/moon-landing/pull/11112 (internal links).

This PR adds support for the gguf key in ModelInfo. Also documenting it as supported in expand[]= parameter. Note that it was supported as soon as the server-side change is made, this PR simply makes it better documented.

Example:

>>> from huggingface_hub import model_info
>>> model = model_info("MaziyarPanahi/solar-pro-preview-instruct-GGUF", expand="gguf")
>>> model.gguf
{'total': 22140032000, 'architecture': 'llama', 'context_length': 4096, 'quantize_imatrix_file': './solar-pro-preview-instruct-GGUF_imatrix.dat', 'chat_template': "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"}

@hanouticelina
Copy link
Contributor

Seems like a concurrent PR with @hlky's #2544 😄

Copy link
Contributor

@hanouticelina hanouticelina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 thanks!

@Wauplin Wauplin merged commit a9adf8f into main Sep 16, 2024
18 checks passed
@Wauplin Wauplin deleted the add-gguf-as-expand-parameter branch September 16, 2024 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants