Fix reformer CI #21254
Fix reformer CI #21254
Conversation
fix ReformerForMaskedLM doc example
| >>> tokenizer.add_special_tokens({"mask_token": "[MASK]"}) # doctest: +IGNORE_RESULT | ||
| >>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt") | ||
| >>> # resize model's embedding matrix |
There was a problem hiding this comment.
This tiny model checkpoint has some issue. Using fast tokenizer, we get vocab size 1000, while using slow tokenizer, it is 320 (something like that).
This is due to the fact the tiny model creation (the old version) sometimes fail to convert a fast tokenizer to slow tokenizer, and it's not clear to me if the slow tokenizer was uploaded separately after the creation, or if there is something strange and the creation gives a slow tokenizer anyway but with smaller vocab size.
There was a problem hiding this comment.
(it's would be better to eventually use the v2 of tiny model creation script though for doctest if necessary)
| >>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt") | ||
| >>> # resize model's embedding matrix | ||
| >>> model.resize_token_embeddings(new_num_tokens=model.config.vocab_size+1) # doctest: +IGNORE_RESULT |
There was a problem hiding this comment.
resize to avoid out of vocab error after the PR #21199 (as it loads the fast tokenizer via AutoTokenizer, which has 1000 tokens as in model config)
| >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) | ||
| >>> tokenizer.decode(predicted_token_id) | ||
| 'it' | ||
| >>> predicted_token = tokenizer.decode(predicted_token_id) |
| >>> outputs = model(**inputs, labels=labels) | ||
| >>> round(outputs.loss.item(), 2) | ||
| 7.09 | ||
| >>> loss = round(outputs.loss.item(), 2) |
| >>> predicted_class_id = logits.argmax().item() | ||
| >>> model.config.id2label[predicted_class_id] | ||
| 'LABEL_0' | ||
| >>> label = model.config.id2label[predicted_class_id] |
There was a problem hiding this comment.
output is random, as this checkpoint is not seq. classification model.
| >>> labels = torch.tensor(1) | ||
| >>> loss = model(**inputs, labels=labels).loss | ||
| >>> round(loss.item(), 2) | ||
| 0.68 |
There was a problem hiding this comment.
output is random, as this checkpoint is not seq. classification model.
|
The documentation is not available anymore as the PR was closed or merged. |
What does this PR do?
Some fixes are required for doctest after #21199. See comments in the review.