AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi, reconnaissance tools

AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi and detection of network reconnaissance tools output. Many bugs fixed and small improvements were performed.

Changes

• [Cryptocurrency + Tools] launch by default + remove old Bitcoin module. [Terrtia]
• [Keys module] detect public key. [Terrtia]
• [Tools detection] add tool detection module. [Terrtia]
• [Cryptocurrency, RegexTracker] update cryptocurrency list + fix: RegexTracker typo. [Terrtia]
• [Cryptocurrency] add private_key entry + fix dash regex. [Terrtia]
• [Cryptocurrency] add new Cryptocurrency module. [Terrtia]
• [Tracker] add optional description field. [Terrtia]

Fix

- [Tool] fix searchsploit regex. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] fix loop. [Terrtia]
- [url_prefix] add root blueprint, fix:#403. [Terrtia]
- [TermTracker] fix performance: disable token stats. [Terrtia]
- [SQL module] fix typo. [Terrtia]

# Other

- Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia]
- Merge pull request #417 from andurin/master. [Alexandre Dulaunoy]

  Fix pybgpranking dependency in requirements
- Fix pybgpranking dependency in requirements. [Hendrik]

  Relates #334
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #404 from WimpyMan/master. [Thirion Aurélien]
- LAUNCH.py: Added execution of script IPAddress.py. [Bastien Schils]
- IPAddress.py: use ipaddress module. [Bastien Schils]

  Improved readability, maintainability and use of standard module
- Modules.cfg: Minor: Added \n to seperate sections. [WimpyMan]
- Config.cfg.sample: Improved example for IP module. [WimpyMan]

  By default, the list of networks to monitor is now empty.
  The previous value is now given as example.
- Added: IP matching module. [Bastien Schils]
- Merge pull request #411 from krial057/patch-1. [Alexandre Dulaunoy]

  Fixed some typos
- Fixed some typos. [krial057]

  Fixed some typos in the readme
- Merge pull request #408 from stamparm/master. [Thirion Aurélien]

  Adding more tools
- Adding more tools. [Miroslav Stampar]
- Merge pull request #407 from stamparm/patch-1. [Thirion Aurélien]

  Covering special cases (on pastebin)
- Covering special cases (on pastebin) [Miroslav Stampar]

  There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier
- Merge pull request #406 from stamparm/master. [Thirion Aurélien]

  Adding tool regexes
- Adding tool regexes. [Miroslav Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #405 from stamparm/master. [Thirion Aurélien]

  Enforcing Base58 check on Litecoin and Dash addresses
- Enforcing Base58 check on Litecoin and Dash addresses. [Miroslav
  Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #401 from stamparm/master. [Thirion Aurélien]

  Improvement of crypto-address regexes (lesser FPs)
- Improvement of crypto-address regexes (lesser FPs) [Miroslav Stampar]
- Merge pull request #398 from stamparm/master. [Thirion Aurélien]

  Implementation for different cryptocurrencies
- Implementation for different cryptocurrencies. [Miroslav Stampar]
- Merge pull request #396 from stamparm/master. [Thirion Aurélien]

  Improvement of SQLi detection
- Improvement of SQLi detection. [Miroslav Stampar]

AIL Framework version 2.2 released with refactoring of term tracking

AIL Framework version 2.2 released with refactoring of term tracking

Changes

• [API] add Tracker documentation. [Terrtia]
• [Tracker] add more info. [Terrtia]
• [update] add v2.2 update + fix default update + fix Empty
  Item.get_tags() + add new LAUNCHER options. [Terrtia]
• [tracker] add missing btn. [Terrtia]
• [trackers] filter trackers list by type + minor fix. [Terrtia]
• [UI sparklines] sparklines: fix + factory. [Terrtia]
• [merge] master. [Terrtia]
• [UI term tracker] refractor term management: trackers list + show
  trackers + add new trackers. [Terrtia]
• [api] add endpoint: get tracked item_id by uuid and daterange.
  [Terrtia]
• [api] add endpoint: delete tracker term (regex/set/word) [Terrtia]
• [Term Tracker] refractor term tracker word/set/regex modules + remove
  old modules. [Terrtia]
• [Term tracker] add term tracker module (word + set) + API: add new
  term to track (word + set + regex) [Terrtia]
• [term] refractor + add new tracked word/set. [Terrtia]
• [README] add link to API documentation. [Thirion Aurélien]
• [tests API] use argv api key. [Terrtia]
• [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail
  metadata + items list. [Terrtia]

Fixes

• [sparkline] datatable drawing. [Terrtia]
• [d3 graph] fix script path. [Terrtia]
• Add missing file. [Terrtia]
• [d3 js plugin] [Terrtia]
• [template] add trackers. [Terrtia]
• [Term Tracker module] chg module flow. [Terrtia]
• [BankAccount] fix #385. [Terrtia]
• [API doc] get bitcoin metadata + list of items. [Terrtia]
• [UI caching] avoid domain archive caching. [Terrtia]
• Typo. [Terrtia]
• [UI caching] fix: #373 avoid screenshot caching. [Terrtia]

(free) Trainings

• Prague, Thursday, 19 Sep 2019 09:00 : https://en.xing-events.com/CLTDKUU.html
• Luxembourg, Monday, 23 Sep 2019 10:00 : https://en.xing-events.com/JDVIRXW.html

AIL Framework version 2.1 released with an API

AIL Framework now includes an advanced API to query the items and information within an AIL instance. You can query items (e.g. such as pastes, crawled website content), meta-data (tag). The API allows to submit item to be processed by the AIL instance. More API endpoints will be added in the next releases.

AIL Framework version 2.0 released including a user management functionality, many bugs fixed and improvements

v2.0 (2019-07-05)

Changes

• [helper] dump crawler history by daterange. [Terrtia]
• [UI submit items] bootstrap 4 migration. [Terrtia]
• [Flask login] add brute force protection + log login errors. [Terrtia]
• [helper] generate self signed certificates. [Terrtia]
• [Flask server] https support + create self signed certificate.
  [Terrtia]
• [user_management 2.0] add update scripts + fix create_default_user.
  [Terrtia]
• [UI user_management] user_role acl: hide admin panel. [Terrtia]
• [UI user_management] incorrect passwords: display errors. [Terrtia]
• [user_management endpoint] check user roles + add 503 template.
  [Terrtia]
• [UI dashboard + search] bootstrap 4 migration: dashbaord + fix search
  input. [Terrtia]
• [UI crawler, show_domain] domain history: remove target blank.
  [Terrtia]
• [UI crawler, show_domain] add domain history list + navigation.
  [Terrtia]
• [user_managemant] clean code + check password and email length.
  [Terrtia]
• [user_management UI] add admin section: edit + create users. [Terrtia]
• [user_management UI] edit my_profile + renew api tokens. [Terrtia]
• [user_management] create default admin user (temp passwd save in
  AIL_HOME) + change password UI + logout UI + create random password.
  [Terrtia]
• [restapi] add rest api authentification + create default user.
  [Terrtia]
• [user_management] add user role_management. [Terrtia]
• [user_management] create + check user password. [Terrtia]
• [UI] add basic user management. [Terrtia]

Fix

• [items submit UI] fix tags dropdown. [Terrtia]
• [helper dump_crawler] fix files not found. [Terrtia]
• [helper dump_crawler] fix empty dict. [Terrtia]
• [MISP export] fix event creation. [Terrtia]
• [UI items_submit] add active tag + fix template name. [Terrtia]
• [UI login/change_password] add missing ail-logo. [Terrtia]
• [Update] clean output. [Terrtia]
• [Update] add default update script. [Terrtia]
• [install_dep] create update current_version. [Terrtia]
• [UI settings] fix toggle_sidebar. [Terrtia]
• [install_dep] create default user. [Terrtia]
• [user_management] fix tokens duplicate + check user_acl_integrity +
  add login errors messages. [Terrtia]
• [server endpoint] unknow users: avoid endpoint enumeration. [Terrtia]
• [TheHive feeder] create_alert: push all items tags TODO check items
  status + add more item metadata. [Terrtia]
• [domain history] fix domain status. [Terrtia]

Other

• Merge pull request #359 from CIRCL/user_management. [Alexandre
  Dulaunoy]

  User/role management

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]

• Update README.md. [Thirion Aurélien]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'user_management' of https://github.com/CIRCL/AIL-
  framework into user_management. [Terrtia]

• Update README.md. [Thirion Aurélien]

  Update install instructions

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

AIL Framework version 1.8 released including many bug fixes, improved PGP handling, ability to export crawled resource.

v1.8 (2019-06-12)

Changes

• [UI crawled domains] Download all domain content (HTML + HAR +
  screenshot) [Terrtia]
• [backend crawler] domains: download 1 archive by crawled (most recent)
  [Terrtia]
• [paste_submit UI] add debug. [Terrtia]
• [template] the "item" project. [Alexandre Dulaunoy]

Fix

• [PgpDump] catch UnicodeDecodeError error. [Terrtia]
• [backend crawler] rename downloaded archive. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [PgpDump] process large pgp blocks. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [UI crawler endpoints] display crawler status + fix #353. [Terrtia]
• [update v1.7] add bs4 requirement. [Terrtia]

Example

Finding relationships between two hidden services via the PGP key published on the website.

AIL Framework version 1.7 released including new correlation features between PGP UIDs and cryptocurrency addresses

Changes

• [correlation] clean files. [Terrtia]
• [update v1.7] update thirdparty. [Terrtia]
• [correlation] add cryptocurrency + refractor correlation. [Terrtia]
• [Bitcoin] map cryptocurrency: bitcoin (DB pivot) [Terrtia]
• [update v1.7] add update scripts. [Terrtia]
• [pgpdump] reprocess tagged items + fix pgpdump. [Terrtia]
• [Update] force update order. [Terrtia]
• [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block
  • avoid keys injection in pgp user_id. [Terrtia]
• [decoded UI] add PgpDump UI + fix hashdecoded js. [Terrtia]
• [decoded items] bootstrap 4 migration. [Terrtia]
• [PgpDump] add PgpDump backend TODO: UI. [Terrtia]
• [crawler] manual/auto crawler: always save screenshots. [Terrtia]
• [crawler] manual/auto crawler: always save screenshots. [Terrtia]

Fix

• [correlation] fix endpoint. [Terrtia]

Other

• Update README.md. [Thirion Aurélien]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]

• Merge pull request #349 from kovacsbalu/fix-paste-encoding. [Thirion
  Aurélien]

  Fix #314

• Use default encoding error from redis. [kovacsbalu]

• Fix #314 Replace char on redis encoding error. Try to use local file
  on other error. [kovacsbalu]

• Merge pull request #350 from kovacsbalu/fix-crawler-rotation. [Thirion
  Aurélien]

  fix: [crawler] rotation

• Hopp, single quote :) [kovacsbalu]

• Fix crawler rotation. [kovacsbalu]

  Before this, crawler processed prioritized onions and after all starts prioritized regular.

AIL Framework version 1.6 released with many bugs fixed and testing improvement

Changes

• [travis] Travis has his own venv where it installs "stuff". Now we
  detect and us it in the launcher. [Steve Clement]
• [travis] Require Python 3.6 to make build faster. [Steve Clement]
• [doc] Some stats on build status/gitter etc. [Steve Clement]
• [hashDecoded] cleanup for the VT message + PEP. [Alexandre Dulaunoy]

Fix

• [faup] fix new return types (bytes to str) [Terrtia]
• [Crawler] force domains/subdomains lower case (rfc4343) [Terrtia]
• [showpaste] fix: #346, avoid None screenshots. [Terrtia]
• [python requirements] rename file. [Terrtia]
• [crawler] typo: domains down. [Terrtia]
• [travis] LAUNCH.sh needs -l to launch... [Steve Clement]
• [travis] Next round of travis fixes. LAUNCH.sh is the only launch
  script needed. chg: [installer] Be way more quiet, watching unzips is
  only fun during development. chg: [installer] Make the arch one +x.
  [Steve Clement]
• [installer] Avoid doing funky sudo pip install moves, it breaks
  python on package managed python installs, if ever, use the --user
  flag. [Steve Clement]
• [travis] Try and require xenial (16.04) and see if it works better.
  [Steve Clement]
• [travis ] There are issues on the 14.04 build system of Travis. This
  fixes it temporarily. [Steve Clement]
• [Onion] typo. [Terrtia]

Other

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
• Chg; [doc crawler] use the torproject torrc. [Terrtia]

AIL Framework version 1.5 released including major improvements in crawler, server management, bootstrap 4 support and many more

AIL Framework version 1.5 released including major improvements in crawler, server management, bootstrap 4 support and many more.

• [UI] Server management. Check for new updates/versions and show background update progress
• [update] Background update process introduced
• [UI] Bootstrap 4 migration started (crawler and tags view)
• Crawler includes new functionalities
  • Port can now be configured
  • Configurable crawling including one-time crawler, regular crawling and type of crawling (e.g. including HAR, screenshots, blacklist management)
• All Items (pastes) are now tagged by date-range
• Many bugs were fixed
• And significant performance improvement in the back-end

AIL Framework version 1.4 released including Tor hidden services crawler and monitoring

AIL Framework version 1.4 released including Tor hidden services crawler and monitoring.

Major new feature:

• Tor hidden service crawler. AIL now includes the ability to crawl and parse output crawled from Tor hidden services.
• Tor onion availability is monitored to detect up and down of hidden services.
• Screenshots are captured and integrated in the analysed output.
• Blurred interface functionality has been added to avoid "burning the eyes" of the security analyst with specific content.
• As the collected information is part of the standard framework, all the AIL modules are available to the crawled hidden services.

New features:

• New export modules for statistics including credentials, phones, banking and TLDs.

Many bugs were fixed.

AIL Framework version 1.3 released including automatic decoding of files from unstructured data

AIL Framework version 1.3 released including automatic decoding of files from unstructured data

New features:

• Detection of IBAN bank accounts are now included
• A cleaner module for decoding files (Base64, hex encoded) from unstructured data
• A new UI for browsing decoded files, their types and relationship

Many bugs fixed and small improvements.