AIL Framework version 3.3 released with an improved item view and many improvements

AIL Framework version 3.3 released with improved item view and many improvements

The AIL Framework version 3.3 released included a refactoring of the item view including a convenient html2text preview of item fetched.
Last origin table is now accessible for all domain crawled to show where the information is coming. Improved yara view in AIL to display the yara rule directly. Various bugs were fixed and bitcoin bech32 addresses are now supported.

Changes

• [UI show Item] refactoring + bootstrap 4 migration. [Terrtia]
• [UI show domain] add last origin table: domain + item. [Terrtia]
• [travis] -> bionic. [Alexandre Dulaunoy]
• [yara trackers UI] add yara trackers, show default yara rule.
  [Terrtia]
• [bitcoin tags + correlation] add bech32 adresses. [Terrtia]

Fix

• [UI correlation graph] fix item links. [Terrtia]
• [UI correlation graph] fix item links. [Terrtia]
• [users management] hide API keys by default + fix delete/edit users.
  [Terrtia]
• [crawler] fix ResponseNeverReceived retry time. [Terrtia]
• [crawler] fix ResponseNeverReceived hanlder, check if splash
  restarted. [Terrtia]
• [Yara tracker] fix mail notification. [Terrtia]
• [Yara tracker] fix mail notification. [Terrtia]
• [RegexTracker] fix missing hostname in mail notification. [Terrtia]
• [Flask server] change default host. [Terrtia]
• [ardb] switch to ail-project ardb fork #38. [Jean-Louis Huynen]
• [Launcher] fix virtualenv loader. [Terrtia]

AIL Framework version 3.2 released with new YARA detection engine with a default set of search patterns, many new correlations type and many bugs fixed.

AIL Framework version 3.2 released with new YARA detection engine with a default set of search patterns, many new correlations type and many bugs fixed.

Changes

• [update] add v3.2. [Terrtia]
• [yara trackers] add debug. [Terrtia]
• [tracker yara] show rule content. [Terrtia]
• [correlation] basic correlation: get_correlation_all_object function.
  [Terrtia]
• [submodule + YARA] add submodule auto update + update v3.1.1.
  [Terrtia]
• [trackers] add yara trackers. [Terrtia]
• [Trackers] email notifications: add tracker description in email
  subject. [Terrtia]
• [username correlation + login redirection] add twitter username
  correlation + redirect to the requested page on login. [Terrtia]
• [Item delete] delete father/child link + remove from domain tree +
  delete all child from the same domain. [Terrtia]
• [feeder pystemon] add debug. [Terrtia]
• [whosh index] add data retention fct. [Terrtia]
• [core] disable Release module. [Terrtia]
• [tor crawler] nyt added. [Alexandre Dulaunoy]
• [API + item library] add a way to check if a user can access the API +
  item import (use item_basic) [Terrtia]
• [black-list onion] keybase added. [Alexandre Dulaunoy]
• [core crawler] add screen script create screen + windows + kill them +
  get list. [Terrtia]
• [crawler proxy] interact with the splash manager API (get list of
  proxies + splash containers + launch them) [Terrtia]
• [launcher] add option to reset UI admin password. [Terrtia]
• [UI dashboard] sort module by name. [Terrtia]
• [LAUNCH] launch telegram module by default. [Terrtia]

Fix

• [install] canevasjs. [Terrtia]
• [UI tags] fix add tags template. [Terrtia]
• [install] travis git unshallow. [Terrtia]
• [trackers yara] remove sleep time. [Terrtia]
• [trackers] typo. [Terrtia]
• [updater] tag subversion. [Terrtia]
• [update v3.1.1] init submodule. [Terrtia]
• [travis] [Terrtia]
• [crawler] fix auto crawler creation. [Terrtia]
• [travis] fetch git last tag. [Terrtia]
• [update thirparty] fix canevasjs install. [Terrtia]
• [install crawler] remove old python requirement. [Terrtia]
• [TheHive feeder] create alert, get item full path. [Terrtia]
• [TheHive feeder] create alert. [Terrtia]
• Typo. [Terrtia]
• Typo. [Terrtia]
• Typo. [Terrtia]
• [username correlation] fix domain correlation. [Terrtia]
• [RegexTracker] fix search regex helper. [Terrtia]
• [regex_helper] [Terrtia]
• [regex tracker] fix timeout. [Terrtia]
• [reset_AIL] add helper + fix soft reset. [Terrtia]
• [Exporter] ad missing file. [Terrtia]
• [update_thirparty] fix popper install. [Terrtia]
• [UI popper version] use popper v1
  (twbs/bootstrap#29842) [Terrtia]
• [MISP auto Export] fix import path. [Terrtia]
• [MISP auto export] fix MISP_TheHive_feeder. [Terrtia]
• [travis] virtualenv. [Terrtia]
• [install] unshallow clone. [Terrtia]
• [requirement] remove old crawler requirements (installed by default)
  [Terrtia]
• [crawler] option to disable screenshots and har. [Terrtia]
• [crawlers lib] missing config file. [Terrtia]
• [create au to crawler] save crawler_delta time. [Terrtia]
• [updater] return boolean. [Terrtia]
• [Credential] fix timeout error. [Terrtia]
• [Credential] fix timeout error. [Terrtia]
• [Credential] force lowercase of mail address. [Terrtia]
• [Credential] fix timeout message. [Terrtia]
• [Mail module] regex timeout. [Terrtia]
• [update v1.5] [Terrtia]
• [Mail module] remove debug. [Terrtia]
• [Mail module] fix dns caching + use redis queue. [Terrtia]
• [UI Decoded items] fix download decoded file. [Terrtia]

Other

• Merge pull request #519 from myasn1k/docker_fixes. [Thirion Aurélien]

  Docker fixes

• Update update_thirdparty.sh. [Pietro Mazzini]

• Remove useless copy. [Pietro Mazzini]

• Uncomment update and explain in docker readme. [Pietro Mazzini]

• Add tz as argument. [Pietro Mazzini]

• Update docker readme (cp) [Pietro Mazzini]

• Update docker readme. [Pietro Mazzini]

• Fix canvas folder name. [Pietro Mazzini]

• Add to docker start crawlers launch. [Pietro Mazzini]

• Update Dockerfile: use ubuntu 18, remove redundant update_thirdparty,
  remove old crawler requirements pip install. [Pietro Mazzini]

• Comment git repo tag in installing_deps, git repo not recognised.
  [Pietro Mazzini]

• LAUNCH update function fail because the folder isn't a git repo,
  comment it. [Pietro Mazzini]

• Merge pull request #22 from C00kie-/master. [Alexandre Dulaunoy]

  allow to configure binding interface

• Update Flask_server.py. [C00kie-]

• Update core.cfg.sample. [C00kie-]

• Typo fixed as ail repo is now lower-case. [Alexandre Dulaunoy]

  Typo fixed as ail repo is now lower-case

• Merge pull request #16 from ChilliSec/patch-1. [Thirion Aurélien]

  Update HOWTO.md

• Update HOWTO.md. [ChilliSec]

• Merge branch 'master' of https://github.com/ail-project/ail-framework.
  [Terrtia]

• Merge pull request #497 from ronaldtf/fix-case-thehive. [Thirion
  Aurélien]

  Fix create case with TheHive

• Fix create case with TheHive. [Ronald Teijeira Fernandez]

AIL Framework version 3.1 with new crawling with cookies, Telegram username correlation, new external feeders (e.g. Twitter) and many improvements

AIL framework v3.1 released

AIL framework v3.1 has been released including many new features such as:

• Crawling website with a set of cookies collected from a browser (allowing to crawl website password protected or alike)
• An extraction module to find Telegram users
• New correlation engine for username starting with Telegram
• Improve timeout on various modules
• New JSON importer to be used with new external feeder. A first Twitter feeder and url-feeder is available to feed specific tweets and discovered urls in AIL.

The AIL project moved into a dedicated project to allow new contributions and project within the AIL project scope. If you want to join us and contribute new modules or specific feeders, don't hesitate to contact us.

Complete changelog

• [update] add update v3.1 + install crawler python requirements by
  default. [Terrtia]
• [UI correlation] add username correlation card. [Terrtia]
• [UI correlation] add username correlation graph. [Terrtia]
• [UI correlation] add username correlation: daterange pages. [Terrtia]
• [telegram module] add new tag: telegram invite code. [Terrtia]
• [telegram backend] add username correlation + save invite hash.
  [Terrtia]
• [importer] add map twitter id - item id, add parents link between
  twitter and url extracted. [Terrtia]
• [travis install] shallow clone. fetch tags + commit id. [Terrtia]
• [travis install] shallow clone. fetch tags. [Terrtia]
• [install] debug travis. [Terrtia]
• [README] update gitter link. [Terrtia]
• [import er url_extract] add item twitter parent. [Terrtia]
• [importer url_extract] fix item id. [Terrtia]
• [importer] add url_extract importer. [Terrtia]
• [API json import] take list as imput. [Terrtia]
• [core import] add AIL JSON import format, API endpoint added (AIL
  feeders) [Terrtia]
• [update doc] update doc install + logo + fix updater. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [update Readme logo + links] [Terrtia]
• [add new logo] [Terrtia]
• [AIL logo UI] [Terrtia]
• [doc] add cookiejar screenshot. [Terrtia]
• [Crawler] default docker memory usage. [Terrtia]
• [MISP export] export domain as domain-crawled object. [Terrtia]
• [domain explorer UI] add shortcut button: misp export + correlation.
  [Terrtia]
• [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix
  screenshot duplicate. [Terrtia]
• [Crawler core + UI] crawler lua: handle retry + fix cookie loader and
  selector. [Terrtia]
• [cookiejar UI] add cookiejar + show all. [Terrtia]
• [crawler cookies] use cookiejar. [Terrtia]
• [crawler - cookies] add/show/select cookies. [Terrtia]
• [crawler] add cookies list by user/global, save cookies from file +
  dict(name, value), TODO: API + handle errors. [Terrtia]
• [crawler] bypass login: use cookie provided by user and accept cookie
  from server + refractor. [Terrtia]
• [Splash Crawler] use cookies to bypass login. [Terrtia]
• [UI] bump jquery to 3.4.1. [Terrtia]
• [import_dir] fix is_gzip test, use magic number. [Terrtia]
• [MISP Importer] files: handle missing sha1/sha256 attributes + fix
  Items and Screenshots dir. [Terrtia]
• [Updater] relauch updater on change (git pull) [Terrtia]

Fixes

• [Mail module] replace signal by multiprocessing
  (https://docs.python.org/3.4/library/signal.html#execution-of-python-
  signal-handlers) [Terrtia]
• [Mail] debug signal. [Terrtia]
• [Mail] debug. [Terrtia]
• [Mail module] remove test time.wait. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [telegram module] typo. [Terrtia]
• [telegram module] fix tagging. [Terrtia]
• [telegram module] check username length. [Terrtia]
• [telegram module] remove debug. [Terrtia]
• [Mails] regex timeout. [Terrtia]
• [Mails] change module output. [Terrtia]
• [Mails] remove print + test. [Terrtia]
• [Mails] refactor Mail module. [Terrtia]
• [urlextract importer] fix parent map + replace set JSON queue by list.
  [Terrtia]
• [import urlextract parent] fix typo. [Terrtia]
• [travis install] shallow clone. fetch missing commit id. [Terrtia]
• [Mails regex timeout] reduce default timeout. [Terrtia]
• [Mails dns resolver] update timeout exception. [Terrtia]
• [Mails] typo. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [urlexport importer] fix item name + redis config. [Terrtia]
• [urlexport importer] item_id, force str type. [Terrtia]
• [urlexport importer] add missing import. [Terrtia]
• [urlextract importer] class name. [Terrtia]
• [importer] fix typo. [Terrtia]
• [Global] extend ungzip error catching, catch invalid compressed file.
  [Terrtia]
• [json import API] remove list input. [Terrtia]
• [install create default user] default passwd file: add missing new
  line. [Terrtia]
• [installer] get last git version. [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme logo size] [Terrtia]
• [show item UI] return 404. [Terrtia]
• [Cookiejar UI] fix typo. [Terrtia]
• [Flask session cookie name] add uuid to cookie name. [Terrtia]
• [crawler] typo. [Terrtia]
• [Crawler splash ResponseNeverReceived] add retry. [Terrtia]
• [crawler] error catcher. [Terrtia]
• [MISP export UI] fix input: name + value overwrite. [Terrtia]
• [update thirdparty] update taxonomies. [Terrtia]
• [crawler] cleanup. [Terrtia]
• [PgpDump parser] remove header comment (rfc4880) + remove empty lines.
  [Terrtia]
• [Pgp Dump] remove tool version. [Terrtia]
• [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller.
  [Terrtia]
• [pgpdump] fix subtype save. [Terrtia]
• [Updater] force updater update. [Terrtia]
• [Updater] fix current_tag parser. [Terrtia]
• [import_dir] remove special characters. [Terrtia]
• [import_dir] remove dir whitespaces #475. [Terrtia]

Other

• Merge pull request #492 from sunil3590/master. [Thirion Aurélien]

  crawler_time -> crawler_delta

• Crawler_time -> crawler_delta. [Sunil D S]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien]

  fix: [crawler] error catcher

• Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien]

  Crawler v2 - Add cookiejar - use cookie to bypass login form

• Merge branch 'master' into crawler_v2. [Terrtia]

AIL Framework version 3.0 with full MISP format export/import and small improvements

AIL Framework version 3.0 with full MISP format export and import has been released. AIL users can now export a set of selected items as a MISP event including objects (items, decoded, screenshot, pgp...), correlations and metadata. There is an import to allow analyst to keep a specific set of analysis in AIL for moving across AIL instances. This release also includes support for authentication of the SMTP server (thanks to Mike Peters for the contribution). Multiple bugs fixed and small improvements.

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

This release of AIL includes a major security fix for CVE-2020-8545. The security vulnerability was in the handler of the global feed which could allow malicious feed providers to overwrite and potentially, execute Python code in the environment. This release also includes various bugs fixed. We urge users to upgrade as soon as possible.

Changes

• [domain explorer] domains explorer v2, filter domains by daterange.
  [Terrtia]

Fix

• [IPAddress] catch empty config error. [Terrtia]
• [Global: already saved filename] save updated + filter duplicated
  items. [Terrtia]
• [Global: filename provided by all feeders] avoid path tranversal.
  [Terrtia]
• [Domain explorer UI] fix daterange pagination links. [Terrtia]
• [Tag core] check if item_date type is an integer. [Terrtia]

AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed

AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed.

There is also a "Practical Darkweb and criminal Blockchain monitoring using AIL (Framework for Analysis of Information Leaks) - free Training/Workshop" hosted by CIRCL (Luxembourg) which will take place the 20th Feb 2020. Registration link.

Next release

In the next release of AIL, only Python 3.6 and upper will be supported.

Changes

• [domain core + UI] add domain explorer v1. [Terrtia]
• [test api] increase import timeout. [Terrtia]
• [UI + core] tag decoded items, fix: #455. [Terrtia]
• [UI item tags search] refractor: search item tag by object, use new
  functions. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]

Fix

• [MISP export] force pymisp version. [Terrtia]
• [tag core] typo. [Terrtia]
• [tag core] item date type. [Terrtia]
• [Tag core] add tag, update tag last seen. [Terrtia]
• [Flask server + cookie session] chg default cookie name (also use port
  number) + add Flask port number to config. [Terrtia]
• [expand btc adress] filter empty addr fields. [Terrtia]
• [UI navbar] chg icon decoded tags. [Terrtia]
• [UI show item min] fix empty modal. [Terrtia]
• Change name popper.js-1.14.3 to popper-core-1.14.3. [mangelft]
• [UI] screenshot url. [Terrtia]

Other

• Merge pull request #453 from mangelft/master. [Thirion Aurélien]

  fix: change name popper.js-1.14.3 to popper-core-1.14.3

• Merge pull request #450 from CIRCL/tags_v2. [Thirion Aurélien]

  Tags v2

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released includes a major refactoring of the tagging system. Tagging speed has been significantly improved. Tags can now be used with items, domains and images and added in the objects. A new feature to search by tags has been also added. Multiple bugs were fixed.

Detailed change logs (v2.7 (2020-01-13))

Changes

• [UI domain] add input: show domain by name. [Terrtia]
• [tags blueprint] clean code. [Terrtia]
• [update v2.7] sort domain full_onion_up and full_regular_up. [Terrtia]
• [UI tags] search domains by tags. [Terrtia]
• [core + UI] search domain by tags. [Terrtia]
• [Update v2.7] add update v2.7 scripts. [Terrtia]
• [tags UI] edit object tags (delete tags) [Terrtia]
• [Tag core] objects tagging, Part 2/2 TODO: UI tags domain + screenshot
  • object rename paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2 TODO UI (tags) + rename
  paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2. [Terrtia]
• [README] remove top terms. [Terrtia]
• [correlation graph UI] add json error handler + add loading status.
  [Terrtia]
• [UI correlation graph + UI domain] correlation screenshot: show img in
  toolip + show hash in ShowDomain TODO: pixelate images. [Terrtia]

Fix

• [UI tags] fix domain links. [Terrtia]
• [UI term] remove deprecated trending charts, fix #446 #447. [Terrtia]
• [Crawler] typo. [Terrtia]
• [UI decoded item] sort mimetype. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain map. [Terrtia]
• [UI showDomain] fix screenshot accordeon. [Terrtia]

Other

• Merge pull request #449 from CIRCL/tags_v2. [Alexandre Dulaunoy]

  Tags v2 - Tagging system refractoring

• Merge branch 'master' into tags_v2. [Terrtia]

• Update README.md. [Thirion Aurélien]

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes. Thanks to the enforce project for the feedback during the training. New features were based on constructive remarks from the users.

New and Improvements

• [slides] source code added. [Alexandre Dulaunoy]
• [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update. [Terrtia]
• [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status. [Terrtia]
• [Domain + Date] get domain up range + get date days and months by daterange. [Terrtia]
• [Domain] get all/by month domains up. [Terrtia]
• [API] get domain metadata (minimal) [Terrtia]
• [UI correlation graph] tooltip: show domain tags. [Terrtia]
• [UI correlation graph] popover: add loading status + chg css. [Terrtia]
• [correlation UI] add basic popover. [Terrtia]
• [slide] update slide. [Terrtia]
• [pgpdump] add debug. [Terrtia]
• Linked TOR installation instruction in the README. [Sami Mokaddem]
• [Onion] add discovery queue. [Terrtia]
• [Showpaste] check if tags are safe (img) + fix domain link. [Terrtia]
• [crawler dashboard UI] add UP/Down domains url. [Terrtia]

Bugs fixed

• [UI correlation graph] typo. [Terrtia]
• [UI showDomain] fix down domain history. [Terrtia]
• [Domain] domain was up. [Terrtia]
• [Domain] is_domain_up. [Terrtia]
• Typo. [Terrtia]
• [Update] force manual update, fix #443. [Terrtia]
• [UI] fix show paste modal. [Terrtia]
• [screenshot canevas + domain link] fix item domain link + screenshot
  canevas: chg colors and icons for unsafe tags. [Terrtia]
• [PgpDump] catch bs4 error. [Terrtia]
• Placed Tor installation instruction in the installation section. [Sami Mokaddem]
• [Update v2.4] fix empty set. [Terrtia]
• [Update v2.4] fix empty set. [Terrtia]
• [Item lib] fix import. [Terrtia]
• [Paste submit] fix tags unpack. [Terrtia]
• [Show Domain UI] fix screenshot link, fix #431. [Terrtia]
• [Update] filter invalid tags. [Terrtia]

AIL Framework version 2.5 released with improved correlation and experimental support for MISP modules

AIL Framework version 2.5 released.

AIL Framework version 2.5 released with correlation and experimental support for MISP modules. The correlation is now improved to add correlation between PGP, cryptocurrencies, pastes, decoded values against any items in AIL framework. The correlation interface has been redesigned to allow filtering per type (pastes, crawled) and also limit the number of correlations. The MISP modules is still very early but the objective is gain from all MISP modules expansion within AIL. We introduced an experimental module to automatically expand BTC transactions from addresses seen in AIL and pivot to new correlations. New roles were added (read_only and users without) to improve the profile of the various AIL users.

The improvements were designed with some requirements from the ENFORCE project to better support law-enforcement usage.

AIL Framework version 2.4 released with improved crawled domain correlation (cryptocurrency, pgp keys, decoded...)

AIL version 2.4 released

AIL version 2.4 has been released including the following new features:

• Improved crawled domain correlation to correlate such domain via cryptocurrency addresses, PGP keys UID, decoded hash content
• Screenshot of crawled item can be selected from the UI
• Crawled domain and port are now properly supported
• Tagging functionality added to domain crawled
• Configuration files have been moved to a coherent directory
• Documentation of the code improved
• Various bugs fixed and small improvements