Skip to content

Start incorporating type checker to GUI file #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ThePhilgrim merged 11 commits into from
Mar 27, 2021
Merged

Start incorporating type checker to GUI file #47

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a88ec6d
Start incorporating type checker to GUI file
ThePhilgrim Mar 22, 2021
8b29568
Add more argument types for mypy
ThePhilgrim Mar 22, 2021
048e7d0
Update upwatch.py
ThePhilgrim Mar 27, 2021
fb64e87
Update upwatch.py
ThePhilgrim Mar 27, 2021
ac2f2d9
Update upwatch.py
ThePhilgrim Mar 27, 2021
e598719
Update upwatch.py
ThePhilgrim Mar 27, 2021
542fc92
Update upwatch_gui.py
ThePhilgrim Mar 27, 2021
aa7d16d
Update upwatch.py
ThePhilgrim Mar 27, 2021
e082d8a
Update upwatch.py
ThePhilgrim Mar 27, 2021
cc1bd0f
Specified job_post_dict as JobPost for mypy
ThePhilgrim Mar 27, 2021
c6c0935
Import Tuple and specify upwatch.List for mypy
ThePhilgrim Mar 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,26 @@
# Upwatch
An Upwork webscraper that will notify you of newly published job posts in your field of work

# PyQt5

# MyPy
python3 -m pip install mypy

mypy *.py


# PyQt5 Stub files for MyPy
python3 -m pip install PyQt5-stubs


# black
python3 -m pip install black

black *.py



#LOGIC
#BeautifulSoup
#Requests
#LXML
53 changes: 39 additions & 14 deletions upwatch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,40 @@
import requests
from bs4 import BeautifulSoup # type: ignore
from typing import TypedDict
import requests
import json
import time
import pathlib
from typing import Any, List

# TODO: use TypedDict for these
JsonContent = Any
JobPost = Any
from typing import Optional, List

# !import re # For looking for eventual word counts in job posts & controlling the validity of url input.

# TypedDict for Type checker to handle "Job Posts" key in json_content
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated
JobPost = TypedDict(
"JobPost",
{
"Job Title": str,
"Payment Type": str,
"Budget": str,
"Job Description": str,
"Job Post URL": str,
},
)

# TypedDict for Type checker to handle json_content
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated
JsonContent = TypedDict(
"JsonContent",
{
"Requests URL": str,
"Run on startup": bool,
"Scrape interval": int,
"DBMR": bool,
"Fixed Lowest Rate": int,
"Hourly Lowest Rate": int,
"Ignore no budget": bool,
"Job Posts": Optional[JobPost],
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated
},
) # "Job Posts" can also be "None" . Does it matter?
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated


# TODO: Add to json: user agent
def read_from_json(json_path: pathlib.Path) -> JsonContent:
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated
Expand Down Expand Up @@ -86,6 +110,7 @@ def json_difference_checker(

old_job_urls = [job_post["Job Post URL"] for job_post in json_content["Job Posts"]]

@Akuli Akuli Mar 23, 2021

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually json_content["Job Posts"] can be None, but not in this function. Unfortunately mypy isn't clever enough to notice that, and we need to tell that to mypy manually. See python/mypy#4245

Suggested change
old_job_urls = [job_post["Job Post URL"] for job_post in json_content["Job Posts"]]
assert json_content["Job Posts"] is not None
old_job_urls = [job_post["Job Post URL"] for job_post in json_content["Job Posts"]]

@ThePhilgrim ThePhilgrim Mar 27, 2021

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please specify why Mypy even needs to notice that it cannot be None? We already told mypy that "it's optionally JobPost" This means that it will be 1 or 0, and in this function it will just always be 1. Why does mypy need more info?

@ThePhilgrim ThePhilgrim Mar 27, 2021

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this suggestion is marked as outdated for some reason. I cannot "commit suggestion", and it is not implemented neither in type_checker or main.


old_job_urls.blah_blah_blah_blah()
Comment thread
ThePhilgrim marked this conversation as resolved.
Outdated
new_job_posts = [
job_post
for job_post in job_post_list
Expand Down Expand Up @@ -124,14 +149,14 @@ def job_post_scraper(json_content: JsonContent) -> List[JobPost]:
) # TODO: Figure out how to fetch User Agent on current system.
response.raise_for_status()
break
# except requests.exceptions.HTTPError as errh: # TODO Error messages need to be communicated to user in a different way.
# print("HTTP Error:", errh)
# print("Please try a different URL")
# return
# except requests.exceptions.ConnectionError:
# print("Error Connecting")
# print("Please check you internet connection and try again.")
# return
# except requests.exceptions.HTTPError as errh: # TODO Error messages need to be communicated to user in a different way.
# print("HTTP Error:", errh)
# print("Please try a different URL")
# return
# except requests.exceptions.ConnectionError:
# print("Error Connecting")
# print("Please check you internet connection and try again.")
# return
except requests.exceptions.Timeout:
print("Your request timed out.")
if connection_attempts == 3:
Expand Down
Loading