TorBot Stable Version1.2 #56
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Version 1.1.0_dev
…/torbot into info_feature_v0.0.1
Info_feature_v0.0.1
Fixed requirements.txt by adding missing "="
Fixed the error occurring when given a broken link with -u flag and other minor improvements.
Adding Contributors
PEP8 code and fixed some minor bugs
Updating my branch
To socket code and put it in side of a function. Also took out the Controller module since it wasn't actually being used.
Removed code that wasn't being called and now using regular expressions to validate urls
Instead of just checking if string contains http or https Uses two functions which use regular expressions to match valid urls. One funciton is specifically geared towards onion address and the other is for general url validation.
Using the -s flag will now save the results in a json file within the current working directory.
Takes list of URLs and asychrnously calls HEAD request on each link in the list and test status code for a 200 response. If the response is not 200 or takes longer than 8 seconds, the link is declared dead. Also switched from urllib.request to requests for not only simplicity but thread-safety also.
Was previously showing dead for any status code which isn't 200, so I use raise_for_status function from requests which only raises an error if the status code is an HTTP error status code such as 4xx and 5xx
To use dash -e flag, pass url name like www.url.com and it will try to establish an https connection than http if the secure conneciton fails. If both fail then an error message is printed. If the connection is successful than we search for the onion domain name and the others that were passed with the flag.
KingAkeem
requested changes
Feb 11, 2018
Fixed requirements.txt, I took out the stem module since it's not used anymore. Added requests module also
|
@KingAkeem Thanks for pointing out. I will fix those asap. |
KingAkeem
requested changes
Feb 12, 2018
modules/pagereader.py
Outdated
| @@ -1,17 +1,36 @@ | |||
| import urllib.request | |||
| import urllib.request | |||
There was a problem hiding this comment.
Needs to be switched to requests
modules/pagereader.py
Outdated
| def readPage(site): | ||
| headers = {'User-Agent': | ||
| 'TorBot - Onion crawler | www.github.com/DedSecInside/TorBot'} | ||
| req = urllib.request.Request(site, None, headers) |
There was a problem hiding this comment.
Also needs to be switched to requests
modules/pagereader.py
Outdated
| while (attempts_left): | ||
| try: | ||
| response = urllib.request.urlopen(req) | ||
| page = BeautifulSoup(response.read(), 'html.parser') |
requirements.txt
Outdated
| @@ -1,3 +1,4 @@ | |||
| beautifulsoup4==4.6.0 | |||
| PySocks==1.6.7 | |||
| stem==1.5.4 | |||
There was a problem hiding this comment.
Stem is no longer being used and request requirement should be added for version 2.18.4
|
This PR will be updated once #66 is merged. @KingAkeem |
Fixed `-l` flag for finding live links.
|
@KingAkeem Please review this |
KingAkeem
requested changes
Feb 12, 2018
| - sudo apt-get -y install python3-pip | ||
| - pip3 install bs4 | ||
| - pip3 install -r requirements.txt | ||
| - cd tests |
There was a problem hiding this comment.
Since we are using pytest, whenever you go into the test suite. You only need to run pytest to run all of the tests.
README.md
Outdated
| 5. Save crawl info to JSON file.(Completed) | ||
| 6. Crawl custom domains.(Completed) | ||
| 7. Check if the link is live.(Not Started) | ||
| 4. Built-in Updater.(Completed) |
There was a problem hiding this comment.
Built-in Updater should be number 8 instead of 4.
|
@KingAkeem @agostinelli @agrepravin @leaen Thanks for the awesome work |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#49