Update pip versions in requirements #155 #156
Conversation
requirements.txt, lock.requirements.txt:
* Versions need to be updated to current versions in
gamechanger-data
Reference:
* dod-advana/gamechanger-data#285
* dod-advana#155
Signed-off-by: Tuan T. Pham <[email protected]>
| @@ -1,19 +1,21 @@ | |||
| annoy~=1.17.0 | |||
There was a problem hiding this comment.
@neofob the current set of requirements used is as follows (try to re-run the pip install with these versions and let me know if it works):
annoy~=1.17.0
awscli==1.27.53
bert-extractive-summarizer~=0.5.2
boto3~=1.26.50
click==7.1.2
elasticsearch==7.13.4
#en-core-web-lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.0.0/en_core_web_lg-3.0.0.tar.gz
en-core-web-md @ https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.0.0/en_core_web_md-3.0.0.tar.gz
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz
#faiss-cpu~=1.7.1
#faiss-gpu~=1.7.1
fastapi_utils~=0.2.1
fastapi~=0.69.0
fuzzywuzzy~=0.18.0
gensim==4.1.2
gunicorn~=20.1.0
httptools~=0.1.2
huggingface-hub==0.7.0
matplotlib~=3.4.3
networkx~=2.6.3
nltk~=3.6.5
numpy~=1.19.5
packaging~=21.0
pandas==1.3.4
protobuf~=3.20.0
psycopg2~=2.8.6
pytest~=6.2.5
PyYAML~=5.4.1
redis~=3.5.3
requests~=2.26.0
scikit-learn~=1.2.0
sentence-transformers~=2.2.2
spacy~=3.0.7
tensorflow~=2.5.1
tokenizers~=0.10.3
torch~=1.7.1
tqdm~=4.62.3
transformers~=4.11.3
txtai~=2.0.0
urllib3~=1.22
uvicorn~=0.13.4
uvloop~=0.14.0
wikipedia~=1.4.0
xgboost==1.1.0
debugpy~=1.6.2
mlflow~=1.27.0
psutil~=5.9.1
There was a problem hiding this comment.
@mishoe :
- The good news is that I can install
gamechangermlwith therequirements.txtthat you provide. - The bad news is that I cannot install gamechanger-data requirements that uses it.
I use the "old" requirements.txt that is removed from repo.
What requirements.txt that you use to build?
I have been using gamechanger repo's build.sh script to build the docker images.
This is where the gamechanger-data docker image is built:
https://github.com/dod-advana/gamechanger/blob/main/deploy/docker-compose/build.yaml#L20
Build error from running build.sh:
...
8.993 Collecting click
9.035 Downloading click-8.0.4-py3-none-any.whl (97 kB)
9.117 Collecting jsonschema
9.169 Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
9.378 Collecting pandas
9.385 Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
10.24 Collecting filetype
10.25 Downloading filetype-1.2.0-py2.py3-none-any.whl (19 kB)
10.26 ERROR: Could not find a version that satisfies the requirement frontend (from versions: none)
10.26 ERROR: No matching distribution found for frontend
|
Hey @neofob , can you try to install Reply back after you figure out if this resolved the issue. |
@mishoe : Thanks for the tip. I uninstall pymypdf and reinstall with version It still does not output anything to the output directory. |
Good, looks like we got past the last issue @neofob Amanda is on PTO until 11/21, and will be the best to answer this directly, but let me give you a quick recommendation. This error is in relation to the script expecting being able to call an RDS instance, which I do not believe is a requirement to parse files locally (although you're getting an exception because you can't connect) Can you try an alternate approach for me - once you get everything installed on the GC data side you should be able to try to run the parse pipelines by the approach described in this README (scroll down on this page to see the readme):https://github.com/dod-advana/gamechanger-data/tree/dev/common/document_parser Specifically these instructions. Let me know if this works! |
So, yes, importing Document code itself works.
Also the function, class No, it does not work. |
|
Output from processing one pdf file |
|
@mishoe @drospond @amaruca141 : The code that tries to connect to the DB is from For now, I comment out that line and the command line |
|
@mishoe @drospond @amaruca141 : Now that I can parse the downloaded PDF files to generate the json files. The document in gamechanger-data refers to the function Is it still a part of the work flow to generate the .csv file to be used in the next step, inserting the json files and csv file to Elasticsearch index? |
|
@neofob here is an older version of a topic model for your usage topic_model_20210208.zip |
|
@mishoe : It still fails to load. Do I need some .py codes as well? I unzip the files to the directory The code in |
|
The file Was it intentional? |
|
@drospond @mishoe @amaruca141 : @drospond, I set the env variables for ES as you mentioned in the call. However, the web UI search bar still does not work. Below is the snippet of those env settings inside the gc-web docker. |
|
Hi @neofob , do you get any errors in the browser or backend consoles when trying to run a search? |
Yes, I did get many errors. First of ALL, the modheader extension.
So I DISABLED the modheader so that I can browse the webpage. Now I can see some errors in the backend. NO, the search query does not return any result. Here is a snippet of the command |
requirements.txt, lock.requirements.txt:
gamechanger-data
Reference:
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Ticket(s) or Issue(s):
Type of change
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Screenshot of automated jenkin tests (if applicable):
Checklist:
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules
I have used an autoformatter like lint or blackc
This requires a code review