You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Install banking77 dataset from huggingface: $ git clone [email protected]:datasets/mteb/banking77
I then run:
importtorchfromllm2vecimportLLM2VecfrommtebimportMTEBl2v=LLM2Vec.from_pretrained(
"<path to model>/llama3-llm2vec",
peft_model_name_or_path="<path to model>/llama3-llm2vec-unsup",
attn_implementation="flash_attention_2",
device_map="cuda"iftorch.cuda.is_available() else"cpu",
torch_dtype=torch.bfloat16,
)
MODEL_NAME="llama3-llm2vec"evaluation=MTEB(tasks=["Banking77Classification"])
results=evaluation.run(l2v, output_folder=f"results/{MODEL_NAME}")
with the following environment variables set in my .bashrc:
export HF_HOME="<path to model>/cache/"export HF_DATASETS_CACHE="<path to data>/big_data/hf/"export TRANSFORMERS_CACHE="<path to model>/cache/"export HF_HUB_OFFLINE=1 # 1 means offline.export HF_DATASETS_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
But I get this error:
Error while evaluating Banking77Classification: Couldn't find a dataset script at <path to data>/mteb/banking77/banking77.py or any data file in the same directory.
I'm not sure where I obtain the script: banking77.py or if I am properly following the offline evaluation instructions. Any help would be immensely appreciated.
The text was updated successfully, but these errors were encountered:
Hey sorry https://github.com/Muennighoff/mteb.git@offlineaccess is very outdated at this point. I recommend you to just use the regular MTEB and make the necessary changes to make it work offline (which would be similar to the changes made in https://github.com/Muennighoff/mteb.git@offlineaccess). I think it should be very straightforward. It would be amazing if you could open a PR to allow offline evaluation, too, so we have it in the main mteb. 🙌
Hello, I'm trying to run MTEB on a cluster without internet access, but I am struggling. Here are the following instructions I've followed:
$ pip install mteb
$ !pip install --upgrade git+https://github.com/Muennighoff/mteb.git@offlineaccess
$ git clone [email protected]:datasets/mteb/banking77
I then run:
with the following environment variables set in my .bashrc:
But I get this error:
Error while evaluating Banking77Classification: Couldn't find a dataset script at <path to data>/mteb/banking77/banking77.py or any data file in the same directory.
I'm not sure where I obtain the script:
banking77.py
or if I am properly following the offline evaluation instructions. Any help would be immensely appreciated.The text was updated successfully, but these errors were encountered: