Skip to content

Latest commit

 

History

History
422 lines (354 loc) · 27.3 KB

README.md

File metadata and controls

422 lines (354 loc) · 27.3 KB
Raw

humble

A humble, and fast, security-oriented HTTP headers analyzer






A quick analysis with 'humble'!

"千里之行，始於足下 - 老子"
("A journey of a thousand miles begins with a single step. - Lao Tzu")

"And if you don't keep your feet, there's no knowing where you might be swept off to. - Bilbo Baggins"

Table of contents

Features
Screenshots
Installation & Update
Installation & Maintenance (Docker)
Usage
Advanced Usage (Linux)
Checks: Missing Headers
Checks: Fingerprint Headers
Checks: Deprecated Headers and Insecure Values
Checks: Empty Values
Guidelines included
To-Do
Further Reading
Contribute
Acknowledgements
License

Features

✔️ 54 checks for enabled security-related HTTP response headers.
✔️ 14 checks for missing security-related HTTP response headers (the ones I consider essential).
✔️ 1186 checks for fingerprinting through HTTP response headers.
✔️ 120 checks for deprecated HTTP response headers/protocols or with insecure/wrong values.
✔️ SSL/TLS checks: requires the amazing https://testssl.sh/.
✔️ Browser support references for enabled HTTP security headers: provided by https://caniuse.com/.
✔️ Two types of analysis: brief and detailed, along with HTTP response headers.
✔️ Can exclude specific HTTP response headers from the analysis.
✔️ Can export each analysis to CSV, HTML5, JSON, PDF 1.4 and TXT (and in a filename and path of your choice).
✔️ Can analyze 'raw response files': text files with HTTP response headers and values. Ex: curl option '--dump-header'.
✔️ Highlights experimental headers in each analysis.
✔️ Each detailed analysis may include up to dozens of official links, references and technical articles.
✔️ l10n: can display each analysis, the messages and almost all errors in English or Spanish.
✔️ Saves each analysis, showing at the end the improvements or deficiencies in relation to the last one.
✔️ Can display analysis statistics: either against a specific URL or all of them.
✔️ Can display fingerprint statistics: either against a specific term or the Top 20.
✔️ Can display guidelines: for enabling security HTTP response headers on popular frameworks, servers and services.
✔️ Code reviewed via Bandit, Flake8, pyinstrument, SonarLint, Sourcery and vermin.
✔️ Tested, one by one, on thousands of URLs.
✔️ Tested on Docker 26.1, Kali Linux 2021.1, macOS 14.2.1 and Windows 10 20H2.
✔️ Almost all the code under one of the most permissive licenses: MIT.
✔️ Regularly updated.
✔️ Minimal dependencies required.
✔️ Featured on Artemis, DefectDojo, HackTricks, Kali Linux and OWASP.
✔️ Developed entirely in my spare time, no strings attached: feel free to try it out and integrate it into your projects!.
✔️ And with the approval of several AI 😄!.

Screenshots

.: (Windows) - Brief analysis.

(Windows) - Brief analysis


.: (Linux) - Brief analysis along with HTTP response headers.

(Linux) - Brief analysis along with HTTP response headers


.: (Linux) - Detailed analysis, in Spanish.

(Linux) - Detailed analysis in Spanish


.: (Linux) - Analysis of a "raw response file". Example.

(Linux) - Analysis of a raw response file


.: (Linux) - SSL/TLS checks.

Options used: -f -g -p -U -s --hints

(Linux) - SSL/TLS checks (requires https://testssl.sh/ and Linux/Unix client)


.: (Linux) - List of HTTP fingerprint headers based on a specific term.

(Linux) - List of HTTP fingerprint headers based on a specific term


.: (Linux) - Brief analysis saved as CSV. Example.

(Linux) - Brief analysis saved as CSV


.: (Windows) - Detailed analysis saved as PDF. Example.

(Windows) - Detailed analysis saved as PDF


.: (Linux) - Detailed analysis saved as HTML. Example.

(Linux) - Detailed analysis saved as HTML


.: (Linux) - Brief analysis saved as JSON. Example.

(Linux) - Brief analysis saved as JSON


.: (Linux) - Analysis history file: Date, URL, Enabled, Missing, Fingerprint, Deprecated/Insecure, Empty headers & Total warnings (the four previous totals).

(Linux) - Analysis history file: Date, URL, Missing, Fingerprint, Deprecated/Insecure, Empty headers & Total warnings (the four previous totals)


.: (Linux) - Statistics of the analysis performed against a specific URL.

(Linux) - Statistics of the analysis performed against a specific URL


.: (Linux) - Statistics of the analysis performed against all URLs, in Spanish.

(Linux) - Statistics of the analysis performed against all URLs in Spanish


Installation & update

Note

Python 3.8 or higher is required.

# Install python3 and python3-pip:
# (Windows) https://www.python.org/downloads/windows/
# (Linux) if not installed by default, install them via, e.g. Synaptic, apt, dnf, yum ...
# (macOS) https://www.python.org/downloads/macos/

# Install Git:
# (Windows) https://git-scm.com/download/win
# (Linux) https://git-scm.com/download/linux
# (macOS) https://git-scm.com/download/mac

# Setting up a virtual environment in Python (pending how to do it in Windows)
# '/home/bluesman/humble_venv' is a example path for the virtual environment

$ python3 -m venv /home/bluesman/humble_venv
$ source /home/bluesman/humble_venv/bin/activate
$ cd /home/bluesman/humble_venv/
$ git clone https://github.com/rfc-st/humble.git
$ cd humble
$ pip3 install -r requirements.txt

# Good practice: deactivate the virtual environment after you have finished using 'humble'
$ deactivate

# Activate the virtual environment to analyze URLs again with 'humble'
$ cd /home/bluesman/humble_venv/
$ source /home/bluesman/humble_venv/bin/activate
$ cd humble

# Updating (weekly): activate the virtual environment and from 'humble' folder
$ git pull

# Updating (Release): activate the virtual environment, download the source code
# .zip file of the most recent Asset and unzip it in the 'humble' folder, overwriting files.
https://github.com/rfc-st/humble/releases

Installation & maintenance (Docker)

Note

Python 3.8 will be used to build the image.

# Install Docker, and make sure it's running:
# E.g. (Linux): https://www.kali.org/docs/containers/installing-docker-on-kali/
# E.g. (macOs): https://docs.docker.com/desktop/install/mac-install/
# E.g. (Windows): https://docs.docker.com/desktop/install/windows-install/

# Build the image, providing the TAG as the latest Release of 'humble': '1.42' in this example.
# https://github.com/rfc-st/humble/releases (Windows may require elevated console privileges)
$ docker build -t humble:1.42 .

# Run the analysis specifying the above TAG, along with the specific options for 'humble':
# '-it', required: allocate a pseudo-TTY and keep the input interactive
# '-rm', required: automatically remove the container and associated anonymous volumes when it exits

# (Linux/macOS)
# E.g. Analyze https://facebook with a brief analysis:
$ docker run -it --rm --name humble humble:1.42 /bin/bash -c "python3 humble.py -u https://facebook.com -b"

# (Windows)
# E.g. Analyze https://facebook with a brief analysis:
$ docker run -it --rm --name humble humble:1.42 python3 humble.py -u https://facebook.com -b

# Removing (and untagging) previous images of 'humble' after upgrading to the latest release.
$ docker rmi humble:1.42

Usage

(Windows) $ py humble.py
(Linux)   $ python3 humble.py
(macOS)   $ python3 humble.py

usage: humble.py [-h] [-a] [-b] [-df] [-e [TESTSSL_PATH]] [-f [FINGERPRINT_TERM]] [-g] [-grd] [-if INPUT_FILE] [-l {es}] [-lic] [-o {csv,html,json,pdf,txt}] [-of OUTPUT_FILE]
                 [-op OUTPUT_PATH] [-r] [-s [SKIP_HEADERS ...]] [-u URL] [-ua USER_AGENT] [-v]

'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-12-03

options:
  -h, --help                  show this help message and exit
  -a                          Shows statistics of the performed analysis; if the '-u' parameter is ommited they will be global
  -b                          Shows overall findings; if omitted detailed ones will be shown
  -df                         Do not follow redirects; if omitted the last redirection will be the one analyzed
  -e [TESTSSL_PATH]           Shows TLS/SSL checks; requires the PATH of https://testssl.sh/
  -f [FINGERPRINT_TERM]       Shows fingerprint statistics; if 'FINGERPRINT_TERM' (e.g., 'Google') is omitted the top 20 results will be shown
  -g                          Shows guidelines for enabling security HTTP response headers on popular frameworks, servers and services
  -grd                        Shows the checks to grade an analysis, along with advice for improvement
  -if INPUT_FILE              Analyzes 'INPUT_FILE': must contain HTTP response headers and values separated by ': '; E.g. 'server: nginx'.
  -l {es}                     Defines the language for displaying analysis, errors and messages; if omitted, will be shown in English
  -lic                        Shows the license for 'humble', along with permissions, limitations and conditions.
  -o {csv,html,json,pdf,txt}  Exports analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; csv/json will have a brief analysis
  -of OUTPUT_FILE             Exports analysis to 'OUTPUT_FILE'; if omitted the default filename of the parameter '-o' will be used
  -op OUTPUT_PATH             Exports analysis to 'OUTPUT_PATH'; must be absolute. If omitted the PATH of 'humble.py' will be used
  -r                          Shows HTTP response headers and a detailed analysis; '-b' parameter will take priority
  -s [SKIP_HEADERS ...]       Skips 'deprecated/insecure' and 'missing' checks for the indicated 'SKIP_HEADERS' (separated by spaces)
  -u URL                      Scheme, host and port to analyze. E.g. https://google.com
  -ua USER_AGENT              User-Agent ID from 'additional/user_agents.txt' file to use. '0' will show all and '1' is the default
  -v, --version               Checks for updates at https://github.com/rfc-st/humble

examples:
  -u URL -a                   Shows statistics of the analysis performed against the URL
  -u URL -b                   Analyzes URL and reports overall findings
  -u URL -b -o csv            Analyzes URL and exports overall findings to CSV format
  -u URL -l es                Analyzes URL and reports (in Spanish) detailed findings
  -u URL -o pdf               Analyzes URL and exports detailed findings to PDF format
  -u URL -o html -of test     Analyzes URL and exports detailed findings to HTML format and 'test' filename
  -u URL -o pdf -op D:/Tests  Analyzes URL and exports detailed findings to PDF format and 'D:/Tests' path
  -u URL -r                   Analyzes URL and reports detailed findings along with HTTP response headers
  -u URL -s ETag NEL          Analyzes URL and skips 'deprecated/insecure' and 'missing' checks for 'ETag' and 'NEL' headers
  -u URL -ua 4                Analyzes URL using the fourth User-Agent of 'additional/user_agents.txt' file
  -a -l es                    Shows statistics (in Spanish) of the analysis performed against all URLs
  -f Google                   Shows HTTP fingerprint headers related to the term 'Google'

Advanced usage (Linux)

.: Show only the analysis summary.

$ python3 humble.py -u https://www.spacex.com | grep -A 8 "\!." | sed $'1i \n'

Show only the analysis summary (Linux)

.: Show only the URL, date and analysis summary.

$ python3 humble.py -u https://www.spacex.com | grep -A8 -E "0. Info|\!." | grep -v "^\[1\." | sed 's/[--]//g' | sed -e '/./b' -e :n -e 'N;s/\n$//;tn' | sed '5,6d' | sed '1i\'

Show URL, date and the analysis summary (Linux)

.: Show only the deprecated headers/protocols and insecure values.

$ python3 humble.py -u https://www.spacex.com | sed -n '/\[4/,/^\[5/ { /^\[5/!p }' | sed '$d' | sed $'1i \n'

Show only the deprecated headers/protocols and insecure values (Linux)

.: Check for HTTP client errors (4XX).

$ python3 humble.py -u https://my.prelude.software/demo/index.pl | grep -A1 -B5 'Note : \|Nota : ' --color=never

Check for HTTP client errors (4XX) (Linux)

.: Analyze multiple URLs and save the results as PDFs.

$ datasets=('https://facebook.com' 'https://github.com' 'https://www.spacex.com'); for dataset in "${datasets[@]}"; do python3 humble.py -u "$dataset" -o pdf; done

Analyze multiple URLs and save the results as PDFs

Checks: enabled headers

Check this file.

Checks: missing headers

Check this file.

Checks: fingerprint headers

Check this file.

Checks: deprecated headers/protocols and insecure values

Check this file.

Note

humble tries to be strict: both in checking HTTP response headers and their values; some of these headers may be experimental and you may not agree with all the results after analysis.

And that's OK! 😃; you should never blindly trust the results of security tools: there should be further work to decide whether the risk is non-existent, potential or real depending on the analyzed URL (its exposure, environment, etc).

Checks: empty values

Any HTTP response header.

Guidelines included to enable security HTTP headers

  • Amazon Web Services
  • Angular
  • Apache HTTP Server
  • Cloudflare
  • LiteSpeed Web Server
  • Microsoft Internet Information Services
  • Nginx
  • Node.js
  • Spring
  • WordPress

To-Do

  • Add more Header/Value checks (only security-oriented)
  • A new detailed analysis of all CSP directives/values (W3C Level 2 & 3)
  • Google Style Python Docstrings and documentation via Sphinx

Further reading

https://caniuse.com/
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
https://github.com/search?q=http+headers+analyze
https://github.com/search?q=http+headers+secure
https://github.com/search?q=http+headers+security
https://owasp.org/www-project-secure-headers/
https://securityheaders.com/
https://scotthelme.co.uk/
https://webtechsurvey.com/common-response-headers
https://www.w3.org

Contribute

Thanks for downloading 'humble', for trying it and for your time!.

Acknowledgements

License

MIT © 2020-2024 Rafa 'Bluesman' Faura ([email protected])
Original Creator - Rafa 'Bluesman' Faura ([email protected])