-
Notifications
You must be signed in to change notification settings - Fork 51
Read stanford configs #88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
I think it should be ok @raphaelsty |
|
It seems that my reviews are in Pending state, so they can't be seen at the moment. I have just one small regrade: I wouldn’t categorize this as a warning, as it’s expected behavior. It’s merely informing you that the StanfordNLP model has successfully loaded the weights. pylate/pylate/models/colbert.py Line 268 in f0899ab
Similarly, I believe there should be a clear distinction between informational events and actual warnings. For instance, if there were an issue loading the file and it fell back to a default, that would merit a warning. pylate/pylate/models/colbert.py Lines 294 to 303 in f0899ab
|
|
That is very fair, I don't know why I started using |
f0899ab to
b957372
Compare
b957372 to
d7e868c
Compare
|
@sam-hey I changed some warnings to info |
|
LGTM |
Right now, we are using default values for doc/query length, markers and attending to expansion tokens when reading stanford models.
This causes some issues as highlighted in #85 and also requires the user to specify a lot of information when using a model that is not using default values, as can be seen with the loading of Jina-ColBERT.
This PR simply add the reading process of the
artifact.metadatafile of Stanford NLP models and read markers, lengths and attend to expansion tokens values.As usual, we override those if the user feed values to the init of the model. Also changed the
attend_to_expansion_tokensparameter to match the other (None by default and override at the end).