add Pipfile and Pipfile.lock for use with pipenv

[mfekadu]

csai-scraping uses pipenv and it is just simply intuitive.

Let's set that up here too

It should make issues like #82 less common and easier to resolve/communicate too

Changes needed

• Pipfile
• Pipfile.lock
• udpates to the README

Here are 2 nice cheat sheets

• https://gist.github.com/bradtraversy/c70a93d6536ed63786c434707b898d55
• https://medium.com/@bettercallshao/pythons-development-environment-manager-a-practitioner-s-cheat-sheet-55e44e99ac2a

Another guide

• https://realpython.com/pipenv-guide/

Pipfiles are toml files

• so learn toml in Y minutes

[mfekadu]

My process for making the Pipfile

1. pipenv --python 3.6.8

2. pipenv shell

3. ./run.sh

...
ModuleNotFoundError: No module named 'FOOBAR_FIZZBUZZ'

4. pipenv install FOOBAR_FIZZBUZZ

5. repeat 3-4 until everything works

6. manually add package_name = "*" to the other nice-to-have packages

[mfekadu]

Problems

• noticed a "slow locking" after pipenv install pandas. Perhaps pandas has many dependencies itself and thus resolving the dependency graph is difficult?

[mfekadu]

mostly done

api/Pipfile

Lines 1 to 66 in 48336b0

	[[source]]
	name = "pypi"
	url = "https://pypi.org/simple"
	verify_ssl = true
	[dev-packages]
	## mostly used in tests
	requests = "==2.23.0"
	## auto formatter
	### may need to run
	### `pipenv lock --pre`
	### or
	### `pipenv install --pre`
	### to "Allow pre-releases" for black
	black = "==19.10b0"
	## linter
	flake8 = "==3.7.9"
	## testing
	hypothesis = "==5.3.1"
	pytest = "==5.3.4"
	## type-checking
	pyre-check = "==0.0.41"
	## like the Unix `make` but better
	invoke = "==1.4.1"
	[packages]
	# REST API
	## minimal framework
	Flask = "==1.1.1"
	## for security
	Flask-Cors = "==3.0.8"
	## used with flask in deployment
	gunicorn = "==20.0.4"
	## simple JSON (de)serialization
	marshmallow = "*"
	# save to google drive
	PyDrive = "==1.3.1"
	# database
	## mysql client
	mysql-connector-python = "==8.0.18"
	## object-relational mapper
	SQLAlchemy = "==1.3.13"
	# QA.py
	pandas = "==1.0.1"
	fuzzywuzzy = "==0.18.0"
	python-Levenshtein = "==0.12.0"
	# natural language processing
	google-api-core = "==1.16.0"
	google-cloud = "==0.34.0"
	google-cloud-automl = "==0.10.0"
	nltk = "==3.4.5"
	spacy = "==2.2.3"
	scikit-learn = "==0.20.2"
	# misc
	## used to generate settings.yml
	PyYAML = "==5.3"
	## avoid PermissionError: [Errno 13] Permission denied
	## for more context see issue #100
	six = "==1.11.0"
	idna = "==2.6"
	## because google-auth 1.11.2 wants setuptools>=40.3.0
	setuptools = ">=40.3.0"
	[requires]
	python_version = "3.6"

api/Pipfile.lock

Lines 1 to 24 in 48336b0

	{
	"_meta": {
	"hash": {
	"sha256": "fb30d39142d3cc83d8909d9f4f4648a60ac33d4ec3a5a94d8dac7b90ef727a24"
	},
	"pipfile-spec": 6,
	"requires": {
	"python_version": "3.6"
	},
	"sources": [
	{
	"name": "pypi",
	"url": "https://pypi.org/simple",
	"verify_ssl": true
	}
	]
	},
	"default": {
	"blis": {
	"hashes": [
	"sha256:00473602629ba69fe6565108e21957e918cb48b59f5bf2f6bfb6e04de42500cb",
	"sha256:03c368c9716ca814c436550a5f1e02ccf74850e613602519e3941d212e5aa177",
	"sha256:135450caabc8aea9bb9250329ebdf7189982d9b57d5c92789b2ba2fe52c247a7",
	"sha256:1402d9cbb0fbc21b749dd5b87d7ee14249e74a0ca38be6ecc56b3b356fca2f21",

just need to update the readme to look like this