Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #205

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open

Develop #205

wants to merge 30 commits into from

Conversation

gitworkflows
Copy link
Contributor

@gitworkflows gitworkflows commented Dec 26, 2024

User description

(Please add to the PR name the issue/s that this PR would close if merged by using a Github keyword. Example: <feature name>. Closes #999. If your PR is made by a single commit, please add that clause in the commit too. This is all required to automate the closure of related issues.)

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

  • I have read and understood the rules about how to Contribute to this project
  • The pull request is for the branch develop
  • A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
    • I strictly followed the documentation "How to create a Plugin"
    • Usage file was updated.
    • Advanced-Usage was updated (in case the plugin provides additional optional configuration).
    • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
    • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
    • If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
    • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
    • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
    • If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
    • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
    • I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
  • If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
  • Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
  • I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
  • If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
  • If the GUI has been modified:
    • I have a provided a screenshot of the result in the PR.
    • I have created new frontend tests for the new component or updated existing ones.
  • After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

  • If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
  • Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

PR Type

Enhancement, Tests


Description

  • Added comprehensive VirusTotal v3 integration:
    • New VT API v3 mixin class with full API capabilities
    • VT ingestor for sample search and download
    • VT intelligence search analyzer improvements
  • Enhanced phishing analysis capabilities:
    • New phishing form compiler analyzer
    • Form field identification and fake data generation
    • Selenium WebDriver wrapper for page interaction
  • Added Elasticsearch integration:
    • Plugin report indexing and search
    • Data model storage and querying
    • Job aggregation endpoints
  • Improved document analysis:
    • Enhanced URL and macro extraction from docx files
    • Base64 string detection
    • External relationship mapping
  • Added extensive test coverage:
    • VT integration tests
    • Elasticsearch integration tests
    • Phishing analyzer tests
    • API endpoint tests
  • Multiple new analyzer configurations:
    • UrlDNA analyzer
    • NERD analyzer
    • Androguard analyzer
    • LNK Info analyzer

Changes walkthrough 📝

Relevant files
Enhancement
14 files
mixins.py
Add VirusTotal v3 API integration mixin class                       

api_app/mixins.py

  • Added VirusTotalv3BaseMixin class with VT API v3 integration
    capabilities
  • Implemented methods for performing VT API requests, handling responses
    and errors
  • Added functionality for retrieving relationships, behavior summaries
    and IOCs from VT
  • Added support for file scanning, rescanning and report polling
  • +585/-0 
    views.py
    Add job rescan and Elasticsearch query features                   

    api_app/views.py

  • Added rescan functionality for jobs
  • Added Elasticsearch integration for plugin report queries
  • Updated job aggregation endpoints for playbooks, users and TLP
  • Added new API endpoints for plugin report searches
  • +180/-22
    virus_total.py
    Add VirusTotal ingestor implementation                                     

    api_app/ingestors_manager/ingestors/virus_total.py

  • Added VirusTotal ingestor implementation
  • Added functionality to search and download samples from VT
  • Added IOC extraction capabilities
  • Added mock responses for testing
  • +324/-0 
    doc_info.py
    Enhanced document analysis and URL extraction capabilities

    api_app/analyzers_manager/file_analyzers/doc_info.py

  • Added new URL extraction capabilities from docx files
  • Enhanced macro analysis to include Base64 strings
  • Added error handling and improved logging
  • Added functionality to extract external relationships and URLs from
    IOCs
  • +169/-27
    phishing_form_compiler.py
    Added phishing form analysis and compilation analyzer       

    api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py

  • Added new phishing form compiler analyzer
  • Added functionality to compile and analyze phishing forms
  • Added form field identification and fake data generation
  • Added proxy and request handling capabilities
  • +247/-0 
    models.py
    Added data models for threat intelligence storage               

    api_app/data_model_manager/models.py

  • Added base data models for threat intelligence
  • Added models for IP, Domain and File data
  • Added fields for signatures and relationships
  • Added serialization and validation capabilities
  • +206/-0 
    vt3_intelligence_search.py
    Refactored VirusTotal intelligence search analyzer             

    api_app/analyzers_manager/observable_analyzers/vt/vt3_intelligence_search.py

  • Simplified VirusTotal intelligence search implementation
  • Added mock response for testing
  • Improved code organization
  • +152/-31
    tasks.py
    Add Elasticsearch integration for plugin reports                 

    threat_matrix/tasks.py

  • Added typing imports and elasticsearch dependencies
  • Refactored job status references to use STATUSES enum
  • Added new task 'send_plugin_report_to_elastic' for sending reports to
    Elasticsearch
  • +116/-8 
    models.py
    Refactor status handling and add Elasticsearch models       

    api_app/models.py

  • Renamed Status to STATUSES for consistency
  • Added SingletonModel base class for singleton pattern
  • Added LastElasticReportUpdate model for tracking updates
  • Added get_value() method to AbstractReport
  • +64/-19 
    nvd_cve.py
    Add NVD CVE analyzer implementation                                           

    api_app/analyzers_manager/observable_analyzers/nvd_cve.py

  • Added NVD CVE analyzer implementation
  • Added CVE format validation
  • Added mock responses for testing
  • +132/-0 
    0005_alter_domaindatamodel_external_references_and_more.py
    Update data model field configurations                                     

    api_app/data_model_manager/migrations/0005_alter_domaindatamodel_external_references_and_more.py

  • Modified field types for domain, file and IP data models
  • Updated field configurations and defaults
  • +141/-0 
    driver_wrapper.py
    Add Selenium WebDriver wrapper for phishing analysis         

    integrations/phishing_analyzers/analyzers/driver_wrapper.py

  • Added Selenium WebDriver wrapper implementation
  • Added exception handling and retry logic
  • Added methods for page interaction and data extraction
  • +135/-0 
    queryset.py
    Refactor Elasticsearch and data model queries                       

    api_app/queryset.py

  • Updated Elasticsearch client references
  • Fixed data model related queries
  • Updated status references to use STATUSES enum
  • +18/-11 
    classes.py
    Add threat evaluation and data model creation                       

    api_app/analyzers_manager/classes.py

  • Added threat evaluation logic
  • Added data model creation methods
  • Updated status references to use STATUSES enum
  • +62/-2   
    Tests
    9 files
    test_tasks.py
    Add tests for plugin report Elasticsearch integration       

    tests/threat_matrix/test_tasks.py

  • Added test cases for sending plugin reports to Elasticsearch
  • Added test setup with mock data for analyzer, connector and pivot
    reports
  • Added test cases for different report types and configurations
  • Added validation of Elasticsearch document formatting
  • +463/-0 
    test_api.py
    Add tests for job rescan functionality                                     

    tests/api_app/test_api.py

  • Added test cases for job rescan functionality
  • Added tests for observable and sample rescans
  • Added tests for playbook and analyzer rescans
  • Added permission tests for rescan functionality
  • +211/-0 
    test_views.py
    Added pivot configuration view tests                                         

    tests/api_app/pivots_manager/test_views.py

  • Added tests for pivot configuration views
  • Added tests for user permissions and access control
  • Added tests for CRUD operations on pivot configs
  • +159/-4 
    test_mixins.py
    Added tests for VirusTotal v3 analyzer mixins                       

    tests/api_app/test_mixins.py

  • Added tests for VirusTotal v3 mixins
  • Added test cases for request parameters and URI generation
  • Added mock responses for testing
  • +188/-0 
    __init__.py
    Enhanced test framework with file analysis capabilities   

    tests/init.py

  • Added helper methods for file analysis testing
  • Updated test user setup and configuration
  • Added methods for job creation and sample analysis
  • +57/-14 
    test_views.py
    Add analyzer configuration tests                                                 

    tests/api_app/analyzers_manager/test_views.py

  • Added tests for analyzer config CRUD operations
  • Added tests for permissions and authorization
  • Updated status references to use STATUSES enum
  • +155/-2 
    test_nvd_cve.py
    Add NVD CVE analyzer tests                                                             

    tests/api_app/analyzers_manager/observable_analyzers/test_nvd_cve.py

  • Added tests for NVD CVE analyzer
  • Added test cases for valid/invalid CVE formats
  • Added mock response data
  • +118/-0 
    test_doc_info.py
    Add Doc Info analyzer tests                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_doc_info.py

  • Added tests for Doc_Info analyzer
  • Added test cases for follina, macros, CVEs and URLs
  • +112/-0 
    test_views.py
    Add data model viewset tests                                                         

    tests/api_app/data_model_manager/test_views.py

  • Added tests for data model viewsets
  • Added test cases for permissions and authorization
  • Added test data setup
  • +114/-0 
    Configuration changes
    17 files
    0128_analyzer_config_phishing_form_compiler.py
    Add PhishingFormCompiler analyzer configuration migration

    api_app/analyzers_manager/migrations/0128_analyzer_config_phishing_form_compiler.py

  • Added migration for new PhishingFormCompiler analyzer configuration
  • Added parameters for proxy, XPath selectors, and form field matching
  • Added default values for form field matching patterns
  • +396/-0 
    0132_analyzer_config_urldna_new_scan.py
    Add UrlDNA analyzer configuration migration                           

    api_app/analyzers_manager/migrations/0132_analyzer_config_urldna_new_scan.py

  • Added migration for new UrlDNA analyzer configuration
  • Added parameters for API key, scan settings and viewport configuration
  • Added default values for scan parameters
  • +401/-0 
    apps.py
    Add data model manager app configuration                                 

    api_app/data_model_manager/apps.py

    • Added Django app configuration for data model manager
    +5/-0     
    0025_ingestor_config_virustotal_example_query.py
    Added VirusTotal example query ingestor configuration       

    api_app/ingestors_manager/migrations/0025_ingestor_config_virustotal_example_query.py

  • Added new VirusTotal ingestor configuration for example query
  • Added parameters for API key, hours, query and IOC extraction
  • Added migration for ingestor configuration
  • +272/-0 
    0001_initial.py
    Initial data model migrations for threat intelligence       

    api_app/data_model_manager/migrations/0001_initial.py

  • Added initial data model migrations for IETF reports
  • Added models for IP, Domain and File data
  • Added fields for signatures, relationships and metadata
  • +354/-0 
    0133_analyzer_config_urldna_search.py
    Added UrlDNA search analyzer configuration                             

    api_app/analyzers_manager/migrations/0133_analyzer_config_urldna_search.py

  • Added new UrlDNA search analyzer configuration
  • Added parameters for API key and search options
  • Added migration for analyzer configuration
  • +247/-0 
    0129_analyzer_config_phishing_extractor.py
    Added phishing extractor analyzer configuration                   

    api_app/analyzers_manager/migrations/0129_analyzer_config_phishing_extractor.py

  • Added new phishing extractor analyzer configuration
  • Added parameters for proxy and window settings
  • Added migration for analyzer configuration
  • +224/-0 
    0136_alter_analyzerconfig_mapping_data_model_and_more.py
    Updated analyzer configuration and file type support         

    api_app/analyzers_manager/migrations/0136_alter_analyzerconfig_mapping_data_model_and_more.py

  • Updated analyzer config fields for data model mapping
  • Updated supported and not supported filetypes
  • Added migration for analyzer configuration changes
  • +189/-0 
    0120_alter_analyzerconfig_not_supported_filetypes_and_more.py
    Updated analyzer file type support configuration                 

    api_app/analyzers_manager/migrations/0120_alter_analyzerconfig_not_supported_filetypes_and_more.py

  • Updated supported and not supported file types
  • Added new file type choices
  • Added migration for file type changes
  • +180/-0 
    0033_pivot_config_extractedonenotefiles.py
    Added OneNote file extraction pivot configuration               

    api_app/pivots_manager/migrations/0033_pivot_config_extractedonenotefiles.py

  • Added new pivot configuration for OneNote file extraction
  • Added parameters for file field comparison
  • Added migration for pivot configuration
  • +149/-0 
    0126_analyzer_config_nerd_analyzer.py
    Add NERD analyzer configuration                                                   

    api_app/analyzers_manager/migrations/0126_analyzer_config_nerd_analyzer.py

  • Added migration for NERD analyzer configuration
  • Configured analyzer parameters and plugin values
  • +163/-0 
    0035_pivot_config_phishingextractortoanalysis.py
    Add phishing extractor pivot configuration                             

    api_app/pivots_manager/migrations/0035_pivot_config_phishingextractortoanalysis.py

  • Added migration for PhishingExtractorToAnalysis pivot
  • Configured pivot parameters and dependencies
  • +156/-0 
    0124_analyzer_config_androguard.py
    Add Androguard analyzer configuration                                       

    api_app/analyzers_manager/migrations/0124_analyzer_config_androguard.py

  • Added migration for Androguard analyzer configuration
  • Configured analyzer parameters and dependencies
  • +129/-0 
    0054_playbook_config_phishinganalysis.py
    Add phishing analysis playbook configuration                         

    api_app/playbooks_manager/migrations/0054_playbook_config_phishinganalysis.py

  • Added migration for PhishingAnalysis playbook configuration
  • Configured playbook parameters and dependencies
  • +125/-0 
    0052_playbook_config_uris.py
    Add URIs playbook configuration                                                   

    api_app/playbooks_manager/migrations/0052_playbook_config_uris.py

  • Added migration for URIs playbook configuration
  • Configured playbook analyzers and pivots
  • +118/-0 
    0121_analyzer_config_lnk_info.py
    Add LNK Info analyzer configuration                                           

    api_app/analyzers_manager/migrations/0121_analyzer_config_lnk_info.py

  • Added migration for LNK Info analyzer configuration
  • Configured analyzer parameters and file types
  • +120/-0 
    db.py
    Update AWS settings imports                                                           

    threat_matrix/settings/db.py

    • Updated AWS settings imports
    +2/-1     
    Additional files (token-limit)
    101 files
    secrets.py
    ...                                                                                                           

    threat_matrix/secrets.py

    ...

    +30/-24 
    basic_observable_analyzer.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/basic_observable_analyzer.py

    ...

    +105/-0 
    test_classes.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/test_classes.py

    ...

    +33/-8   
    crowdsec.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/crowdsec.py

    ...

    +80/-0   
    serializers.py
    ...                                                                                                           

    api_app/pivots_manager/serializers.py

    ...

    +68/-3   
    strings_info.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/strings_info.py

    ...

    +49/-0   
    extract_phishing_site.py
    ...                                                                                                           

    integrations/phishing_analyzers/analyzers/extract_phishing_site.py

    ...

    +83/-0   
    sample_download.py
    ...                                                                                                           

    api_app/visualizers_manager/visualizers/sample_download.py

    ...

    +88/-0   
    elasticsearch.py
    ...                                                                                                           

    threat_matrix/settings/elasticsearch.py

    ...

    +34/-12 
    0123_basic_observable_analyzer.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0123_basic_observable_analyzer.py

    ...

    +87/-0   
    classes.py
    ...                                                                                                           

    api_app/classes.py

    ...

    +12/-5   
    test_analyzer_extractor.py
    ...                                                                                                           

    tests/api_app/visualizers_manager/passive_dns/test_analyzer_extractor.py

    ...

    +7/-7     
    boxjs_scan.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/boxjs_scan.py

    ...

    +34/-1   
    test_classes.py
    ...                                                                                                           

    tests/api_app/visualizers_manager/test_classes.py

    ...

    +69/-0   
    elastic.py
    ...                                                                                                           

    api_app/serializers/elastic.py

    ...

    +72/-0   
    serializers.py
    ...                                                                                                           

    api_app/playbooks_manager/serializers.py

    ...

    +11/-5   
    nerd.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/nerd.py

    ...

    +68/-0   
    0135_data_mapping.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0135_data_mapping.py

    ...

    +58/-0   
    serializers.py
    ...                                                                                                           

    api_app/analyzers_manager/serializers.py

    ...

    +43/-0   
    0036_alter_extractedonenotefiles_resubmitdownloadedfile_loadfilesameplaybook.py
    ...                                                                                                           

    api_app/pivots_manager/migrations/0036_alter_extractedonenotefiles_resubmitdownloadedfile_loadfilesameplaybook.py

    ...

    +52/-0   
    plugin.py
    ...                                                                                                           

    api_app/serializers/plugin.py

    ...

    +5/-6     
    signals.py
    ...                                                                                                           

    api_app/signals.py

    ...

    +23/-1   
    0004_alter_domaindatamodel_evaluation_and_more.py
    ...                                                                                                           

    api_app/data_model_manager/migrations/0004_alter_domaindatamodel_evaluation_and_more.py

    ...

    +63/-0   
    phishing_extractor.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/phishing/phishing_extractor.py

    ...

    +54/-0   
    admin.py
    ...                                                                                                           

    api_app/data_model_manager/admin.py

    ...

    +61/-0   
    greynoiseintel.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/greynoiseintel.py

    ...

    +43/-0   
    test_strings_info.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_strings_info.py

    ...

    +40/-0   
    0125_update_yara_repo.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0125_update_yara_repo.py

    ...

    +40/-0   
    0064_vt_sample_download.py
    ...                                                                                                           

    api_app/migrations/0064_vt_sample_download.py

    ...

    +53/-0   
    classes.py
    ...                                                                                                           

    api_app/visualizers_manager/classes.py

    ...

    +45/-0   
    dshield.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/dshield.py

    ...

    +53/-0   
    views.py
    ...                                                                                                           

    api_app/pivots_manager/views.py

    ...

    +36/-4   
    serializers.py
    ...                                                                                                           

    api_app/data_model_manager/serializers.py

    ...

    +49/-0   
    0056_download_sample_vt.py
    ...                                                                                                           

    api_app/playbooks_manager/migrations/0056_download_sample_vt.py

    ...

    +38/-0   
    0039_sample_download.py
    ...                                                                                                           

    api_app/visualizers_manager/migrations/0039_sample_download.py

    ...

    +38/-0   
    admin.py
    ...                                                                                                           

    api_app/admin.py

    ...

    +25/-0   
    artifacts.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/artifacts.py

    ...

    +2/-11   
    app.py
    ...                                                                                                           

    integrations/phishing_analyzers/app.py

    ...

    +39/-0   
    compare.py
    ...                                                                                                           

    api_app/pivots_manager/pivots/compare.py

    ...

    +4/-24   
    test_iocextract.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_iocextract.py

    ...

    +33/-0   
    test_serializers.py
    ...                                                                                                           

    tests/api_app/pivots_manager/test_serializers.py

    ...

    +37/-0   
    test_serializers.py
    ...                                                                                                           

    tests/api_app/test_serializers.py

    ...

    +4/-3     
    0051_add_lnk_info_analyzer_free_to_use.py
    ...                                                                                                           

    api_app/playbooks_manager/migrations/0051_add_lnk_info_analyzer_free_to_use.py

    ...

    +34/-0   
    0053_add_androguard_to_free_to_use_analyzers.py
    ...                                                                                                           

    api_app/playbooks_manager/migrations/0053_add_androguard_to_free_to_use_analyzers.py

    ...

    +34/-0   
    test_serializers.py
    ...                                                                                                           

    tests/api_app/data_model_manager/test_serializers.py

    ...

    +44/-0   
    yara_scan.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/yara_scan.py

    ...

    +30/-0   
    analyzer_extractor.py
    ...                                                                                                           

    api_app/visualizers_manager/visualizers/passive_dns/analyzer_extractor.py

    ...

    +15/-14 
    test_crons.py
    ...                                                                                                           

    tests/test_crons.py

    ...

    +4/-4     
    0131_analyzer_config_vt_sample_download.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0131_analyzer_config_vt_sample_download.py

    ...

    +34/-0   
    lnk_info.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/lnk_info.py

    ...

    +37/-0   
    elastic_templates.py
    ...                                                                                                           

    api_app/management/commands/elastic_templates.py

    ...

    +39/-0   
    test_boxjs.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_boxjs.py

    ...

    +37/-0   
    constants.py
    ...                                                                                                           

    api_app/analyzers_manager/constants.py

    ...

    +10/-2   
    pdf_info.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/pdf_info.py

    ...

    +5/-4     
    test_onenote_info.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_onenote_info.py

    ...

    +33/-0   
    models.py
    ...                                                                                                           

    api_app/investigations_manager/models.py

    ...

    +5/-5     
    androguard.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/androguard.py

    ...

    +35/-0   
    maxmind.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/maxmind.py

    ...

    +26/-0   
    test_pdf_info.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_pdf_info.py

    ...

    +34/-0   
    0002_domaindatamodel_resolutions_and_more.py
    ...                                                                                                           

    api_app/data_model_manager/migrations/0002_domaindatamodel_resolutions_and_more.py

    ...

    +38/-0   
    test_classes.py
    ...                                                                                                           

    tests/api_app/test_classes.py

    ...

    +3/-3     
    test_lnk_info.py
    ...                                                                                                           

    tests/api_app/analyzers_manager/file_analyzers/test_lnk_info.py

    ...

    +29/-0   
    0122_alter_soft_time_limit.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0122_alter_soft_time_limit.py

    ...

    +34/-0   
    0137_analyzerreport_data_model_content_type_and_more.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0137_analyzerreport_data_model_content_type_and_more.py

    ...

    +34/-0   
    vt3_sample_download.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/vt/vt3_sample_download.py

    ...

    +33/-0   
    onenote.py
    ...                                                                                                           

    api_app/analyzers_manager/file_analyzers/onenote.py

    ...

    +10/-0   
    0063_singleton_and_elastic_report.py
    ...                                                                                                           

    api_app/migrations/0063_singleton_and_elastic_report.py

    ...

    +39/-0   
    views.py
    ...                                                                                                           

    api_app/analyzers_manager/views.py

    ...

    +17/-2   
    views.py
    ...                                                                                                           

    api_app/playbooks_manager/views.py

    ...

    +4/-1     
    any_compare.py
    ...                                                                                                           

    api_app/pivots_manager/pivots/any_compare.py

    ...

    +13/-8   
    app.py
    ...                                                                                                           

    integrations/malware_tools_analyzers/app.py

    ...

    +10/-5   
    dns.py
    ...                                                                                                           

    api_app/visualizers_manager/visualizers/dns.py

    ...

    +3/-3     
    documents.py
    ...                                                                                                           

    api_app/documents.py

    ...

    +10/-2   
    test_classes.py
    ...                                                                                                           

    tests/api_app/connectors_manager/test_classes.py

    ...

    +3/-3     
    aws.py
    ...                                                                                                           

    threat_matrix/settings/aws.py

    ...

    +7/-2     
    serializers.py
    ...                                                                                                           

    api_app/ingestors_manager/serializers.py

    ...

    +19/-2   
    0138_alter_analyzerreport_data_model_content_type.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0138_alter_analyzerreport_data_model_content_type.py

    ...

    +29/-0   
    abuseipdb.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/abuseipdb.py

    ...

    +12/-0   
    views.py
    ...                                                                                                           

    api_app/data_model_manager/views.py

    ...

    +30/-0   
    celery.py
    ...                                                                                                           

    threat_matrix/celery.py

    ...

    +9/-0     
    load_file.py
    ...                                                                                                           

    api_app/pivots_manager/pivots/load_file.py

    ...

    +11/-2   
    talos.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/talos.py

    ...

    +16/-0   
    urls.py
    ...                                                                                                           

    api_app/data_model_manager/urls.py

    ...

    +22/-0   
    0034_changed_resubmitdownloadedfile_playbook_to_execute.py
    ...                                                                                                           

    api_app/pivots_manager/migrations/0034_changed_resubmitdownloadedfile_playbook_to_execute.py

    ...

    +25/-0   
    urls.py
    ...                                                                                                           

    api_app/urls.py

    ...

    +3/-0     
    dumpplugin.py
    ...                                                                                                           

    api_app/management/commands/dumpplugin.py

    ...

    +9/-2     
    0003_remove_ipdatamodel_ietf_report_and_more.py
    ...                                                                                                           

    api_app/data_model_manager/migrations/0003_remove_ipdatamodel_ietf_report_and_more.py

    ...

    +24/-0   
    enums.py
    ...                                                                                                           

    api_app/data_model_manager/enums.py

    ...

    +23/-0   
    0139_alter_analyzerconfig_mapping_data_model.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0139_alter_analyzerconfig_mapping_data_model.py

    ...

    +22/-0   
    download_file_from_uri.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/download_file_from_uri.py

    ...

    +3/-3     
    abuse_submitter.py
    ...                                                                                                           

    api_app/connectors_manager/connectors/abuse_submitter.py

    ...

    +6/-0     
    test_models.py
    ...                                                                                                           

    tests/api_app/investigations_manager/test_models.py

    ...

    +2/-2     
    queryset.py
    ...                                                                                                           

    api_app/analyzers_manager/queryset.py

    ...

    +8/-0     
    0134_analyzerconfig_mapping_data_model.py
    ...                                                                                                           

    api_app/analyzers_manager/migrations/0134_analyzerconfig_mapping_data_model.py

    ...

    +20/-0   
    celery.py
    ...                                                                                                           

    threat_matrix/settings/celery.py

    ...

    +1/-6     
    __init__.py
    ...                                                                                                           

    threat_matrix/settings/init.py

    ...

    +2/-0     
    signals.py
    ...                                                                                                           

    api_app/pivots_manager/signals.py

    ...

    +17/-0   
    vt3_get.py
    ...                                                                                                           

    api_app/analyzers_manager/observable_analyzers/vt/vt3_get.py

    ...

    +5/-2     
    load_file_same_playbook.py
    ...                                                                                                           

    api_app/pivots_manager/pivots/load_file_same_playbook.py

    ...

    +15/-0   
    fields.py
    ...                                                                                                           

    api_app/data_model_manager/fields.py

    ...

    +19/-0   
    Additional 147 files not shown
    ...                                                                                                           

    Additional 147 files not shown

    ...

    Additional files (token-limit)extra_file_yaml =
    1 files
    exceptions.py
    ...                                                                                                           

    api_app/exceptions.py

    ...

    +7/-1     

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    gitworkflows and others added 26 commits November 8, 2024 16:11
    Signed-off-by: NxPKG <[email protected]>
    Signed-off-by: NxPKG <[email protected]>
    Signed-off-by: NxPKG <[email protected]>
    Signed-off-by: NxPKG <[email protected]>
    * Update compose-tests.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose-tests.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose-tests.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose-tests.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    * Update compose.yml
    
    Signed-off-by: NxPKG <[email protected]>
    
    ---------
    
    Signed-off-by: NxPKG <[email protected]>
    Signed-off-by: gitworkflows <[email protected]>
    upper_case=True,
    lower_case=True,
    )
    logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}")

    Check failure

    Code scanning / CodeQL

    Clear-text logging of sensitive information High

    This expression logs
    sensitive data (password)
    as clear text.

    Copilot Autofix AI 1 day ago

    To fix the problem, we should avoid logging the sensitive information directly. Instead, we can log a message indicating that the fake password was generated without including the actual password in the log. This way, we maintain the functionality of logging the event without exposing sensitive information.

    • Replace the line that logs the fake password with a more generic message.
    • Ensure that no sensitive information is logged in clear text.
    Suggested changeset 1
    api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py

    Autofix patch

    Autofix patch
    Run the following command in your local git repository to apply this patch
    cat << 'EOF' | git apply
    diff --git a/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py b/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    --- a/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    +++ b/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    @@ -95,3 +95,3 @@
             )
    -        logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}")
    +        logger.info("Generated fake password input")
             self.FAKE_TEL_INPUT: str = fake.phone_number()
    EOF
    @@ -95,3 +95,3 @@
    )
    logger.info(f"Generated fake password input {self.FAKE_PASSWORD_INPUT}")
    logger.info("Generated fake password input")
    self.FAKE_TEL_INPUT: str = fake.phone_number()
    Copilot is powered by AI and may make mistakes. Always verify output.
    Positive Feedback
    Negative Feedback

    Provide additional feedback

    Please help us improve GitHub Copilot by sharing more details about this comment.

    Please select one or more of the options
    )

    logger.info(
    f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}"

    Check failure

    Code scanning / CodeQL

    Clear-text logging of sensitive information High

    This expression logs
    sensitive data (password)
    as clear text.

    Copilot Autofix AI 1 day ago

    To fix the problem, we need to ensure that sensitive information, such as passwords, is not logged. We can achieve this by conditionally logging the value based on the input type. Specifically, we should avoid logging the actual value for password fields and instead log a placeholder or a message indicating that a password was provided.

    • Modify the logging statement on line 190 to check if the input type is "password" and log a placeholder instead of the actual value.
    • Ensure that other sensitive information is similarly protected if applicable.
    Suggested changeset 1
    api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py

    Autofix patch

    Autofix patch
    Run the following command in your local git repository to apply this patch
    cat << 'EOF' | git apply
    diff --git a/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py b/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    --- a/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    +++ b/api_app/analyzers_manager/file_analyzers/phishing/phishing_form_compiler.py
    @@ -188,5 +188,10 @@
     
    -            logger.info(
    -                f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}"
    -            )
    +            if input_type.lower() == "password":
    +                logger.info(
    +                    f"Job #{self.job_id}: Sending value [REDACTED] for {input_name=}"
    +                )
    +            else:
    +                logger.info(
    +                    f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}"
    +                )
                 result.setdefault(input_name, value_to_set)
    EOF
    @@ -188,5 +188,10 @@

    logger.info(
    f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}"
    )
    if input_type.lower() == "password":
    logger.info(
    f"Job #{self.job_id}: Sending value [REDACTED] for {input_name=}"
    )
    else:
    logger.info(
    f"Job #{self.job_id}: Sending value {value_to_set} for {input_name=}"
    )
    result.setdefault(input_name, value_to_set)
    Copilot is powered by AI and may make mistakes. Always verify output.
    Positive Feedback
    Negative Feedback

    Provide additional feedback

    Please help us improve GitHub Copilot by sharing more details about this comment.

    Please select one or more of the options

    except Exception as e:
    logging.exception(
    f"Error: {e}. Secret: {secret_name}"

    Check failure

    Code scanning / CodeQL

    Clear-text logging of sensitive information High

    This expression logs
    sensitive data (secret)
    as clear text.

    Copilot Autofix AI about 2 months ago

    To fix the problem, we should avoid logging sensitive information directly. Instead, we can log a generic message that does not include the sensitive secret_name. This way, we still capture the occurrence of an error without exposing sensitive data.

    • Replace the logging statements that include secret_name with more generic messages.
    • Ensure that the functionality of error logging is maintained without exposing sensitive information.
    Suggested changeset 1
    threat_matrix/secrets.py

    Autofix patch

    Autofix patch
    Run the following command in your local git repository to apply this patch
    cat << 'EOF' | git apply
    diff --git a/threat_matrix/secrets.py b/threat_matrix/secrets.py
    --- a/threat_matrix/secrets.py
    +++ b/threat_matrix/secrets.py
    @@ -85,3 +85,3 @@
                 logging.error(
    -                f"Failed retrieving of secret {secret_name}. Error: {e}."
    +                f"Failed retrieving of secret. Error: {e}."
                 )  # lgtm [py/clear-text-logging-sensitive-data]
    @@ -89,3 +89,3 @@
                 logging.error(
    -                f"Error: {e}. Secret: {secret_name}"
    +                f"Error: {e}. Failed to retrieve secret."
                 )  # lgtm [py/clear-text-logging-sensitive-data]
    @@ -93,3 +93,3 @@
                 logging.exception(
    -                f"Error: {e}. Secret: {secret_name}"
    +                f"Error: {e}. Failed to retrieve secret."
                 )  # lgtm [py/clear-text-logging-sensitive-data]
    EOF
    @@ -85,3 +85,3 @@
    logging.error(
    f"Failed retrieving of secret {secret_name}. Error: {e}."
    f"Failed retrieving of secret. Error: {e}."
    ) # lgtm [py/clear-text-logging-sensitive-data]
    @@ -89,3 +89,3 @@
    logging.error(
    f"Error: {e}. Secret: {secret_name}"
    f"Error: {e}. Failed to retrieve secret."
    ) # lgtm [py/clear-text-logging-sensitive-data]
    @@ -93,3 +93,3 @@
    logging.exception(
    f"Error: {e}. Secret: {secret_name}"
    f"Error: {e}. Failed to retrieve secret."
    ) # lgtm [py/clear-text-logging-sensitive-data]
    Copilot is powered by AI and may make mistakes. Always verify output.
    Positive Feedback
    Negative Feedback

    Provide additional feedback

    Please help us improve GitHub Copilot by sharing more details about this comment.

    Please select one or more of the options
    Copy link

    @sourcery-ai sourcery-ai bot left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    The pull request #205 has too many files changed.

    We can only review pull requests with up to 300 changed files, and this pull request has 331.

    Copy link

    coderabbitai bot commented Dec 26, 2024

    Important

    Review skipped

    More than 25% of the files skipped due to max files limit. The review is being skipped to prevent a low-quality review.

    151 files out of 299 files are above the max files limit of 75. Please upgrade to Pro plan to get higher limits.

    You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


    Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

    ❤️ Share
    🪧 Tips

    Chat

    There are 3 ways to chat with CodeRabbit:

    • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
      • I pushed a fix in commit <commit_id>, please review it.
      • Generate unit testing code for this file.
      • Open a follow-up GitHub issue for this discussion.
    • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
      • @coderabbitai generate unit testing code for this file.
      • @coderabbitai modularize this function.
    • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
      • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
      • @coderabbitai read src/utils.ts and generate unit testing code.
      • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
      • @coderabbitai help me debug CodeRabbit configuration file.

    Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

    CodeRabbit Commands (Invoked using PR comments)

    • @coderabbitai pause to pause the reviews on a PR.
    • @coderabbitai resume to resume the paused reviews.
    • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
    • @coderabbitai full review to do a full review from scratch and review all the files again.
    • @coderabbitai summary to regenerate the summary of the PR.
    • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
    • @coderabbitai resolve resolve all the CodeRabbit review comments.
    • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
    • @coderabbitai help to get help.

    Other keywords and placeholders

    • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
    • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
    • Add @coderabbitai anywhere in the PR title to generate the title automatically.

    CodeRabbit Configuration File (.coderabbit.yaml)

    • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
    • Please see the configuration documentation for more information.
    • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

    Documentation and Community

    • Visit our Documentation for detailed information on how to use CodeRabbit.
    • Join our Discord Community to get help, request features, and share feedback.
    • Follow us on X/Twitter for updates and announcements.

    Copy link

    gitguardian bot commented Dec 26, 2024

    ⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

    Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

    🔎 Detected hardcoded secret in your pull request
    GitGuardian id GitGuardian status Secret Commit Filename
    13180230 Triggered Username Password 6ca0a96 tests/auth/test_auth.py View secret
    🛠 Guidelines to remediate hardcoded secrets
    1. Understand the implications of revoking this secret by investigating where it is used in your code.
    2. Replace and store your secret safely. Learn here the best practices.
    3. Revoke and rotate this secret.
    4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

    To avoid such incidents in the future consider


    🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

    Copy link

    deepsource-io bot commented Dec 26, 2024

    Here's the code health analysis summary for commits d7f231f..4d62705. View details on DeepSource ↗.

    Analysis Summary

    AnalyzerStatusSummaryLink
    DeepSource Python LogoPython❌ Failure
    ❗ 402 occurences introduced
    🎯 151 occurences resolved
    View Check ↗
    DeepSource Docker LogoDocker❌ Failure
    ❗ 19 occurences introduced
    🎯 9 occurences resolved
    View Check ↗

    💡 If you’re a repository administrator, you can configure the quality gates from the settings.

    Copy link
    Contributor

    codiumai-pr-agent-free bot commented Dec 26, 2024

    CI Failure Feedback 🧐

    (Checks updated until commit 4d62705)

    Action: dependency-review

    Failed stage: Dependency Review [❌]

    Failure summary:

    The action failed due to security vulnerabilities detected in dependencies:

  • High severity vulnerability in Django 4.2.16: SQL injection vulnerability in HasKey(lhs, rhs) on
    Oracle database
  • Moderate severity vulnerability in Django 4.2.16: Denial-of-service vulnerability in
    django.utils.html.strip_tags()

    The dependency review check failed because these security issues need to be addressed before the PR
    can be merged.

  • Relevant error logs:
    1:  ##[group]Operating System
    2:  Ubuntu
    ...
    
    100:  with:
    101:  repo-token: ***
    102:  ##[endgroup]
    103:  ##[group]Vulnerabilities
    104:  �[1mrequirements/project-requirements.txt » [email protected]�[22m – Django SQL injection in HasKey(lhs, rhs) on Oracle �[31m(high severity)�[39m
    105:  ↪ https://github.com/advisories/GHSA-m9g8-fxxm-xg86
    106:  �[1mrequirements/project-requirements.txt » [email protected]�[22m – Django denial-of-service in django.utils.html.strip_tags() �[33m(moderate severity)�[39m
    107:  ↪ https://github.com/advisories/GHSA-8498-2h75-472j
    108:  ##[error]Dependency review detected vulnerable packages.
    ...
    
    309:  npm/object-inspect: OpenSSF Scorecard Score: 4.9
    310:  npm/object-is: OpenSSF Scorecard Score: 4.4
    311:  npm/object.assign: OpenSSF Scorecard Score: 5.3
    312:  npm/object.entries: OpenSSF Scorecard Score: 4.4
    313:  npm/object.fromentries: OpenSSF Scorecard Score: 4.4
    314:  npm/object.groupby: OpenSSF Scorecard Score: undefined
    315:  npm/object.values: OpenSSF Scorecard Score: 4.4
    316:  npm/prettier: OpenSSF Scorecard Score: 6.6
    317:  npm/react-error-boundary: OpenSSF Scorecard Score: 4
    ...
    
    365:  npm/@types/jest: OpenSSF Scorecard Score: 7
    366:  npm/axios: OpenSSF Scorecard Score: 5.9
    367:  npm/eslint: OpenSSF Scorecard Score: 6.8
    368:  npm/eslint-plugin-import: OpenSSF Scorecard Score: 6.1
    369:  npm/eslint-plugin-jsx-a11y: OpenSSF Scorecard Score: 6
    370:  npm/eslint-plugin-react: OpenSSF Scorecard Score: 5.8
    371:  npm/eslint-plugin-react-hooks: OpenSSF Scorecard Score: 5.2
    372:  npm/prettier: OpenSSF Scorecard Score: 6.6
    373:  npm/react-error-boundary: OpenSSF Scorecard Score: 4
    ...
    
    607:  �[32m+ [email protected]�[39m
    608:  �[32m+ [email protected]�[39m
    609:  �[32m+ [email protected]�[39m
    610:  �[32m+ [email protected]�[39m
    611:  �[32m+ [email protected]�[39m
    612:  �[32m+ [email protected]�[39m
    613:  �[32m+ [email protected]�[39m
    614:  �[32m+ [email protected]�[39m
    615:  �[32m+ [email protected]�[39m
    ...
    
    844:  �[31m- [email protected]�[39m
    845:  �[31m- [email protected]�[39m
    846:  �[31m- [email protected]�[39m
    847:  �[31m- [email protected]�[39m
    848:  �[31m- [email protected]�[39m
    849:  �[31m- [email protected]�[39m
    850:  �[31m- [email protected]�[39m
    851:  �[31m- [email protected]�[39m
    852:  �[31m- [email protected]�[39m
    ...
    
    892:  �[32m+ @types/jest@^29.5.14�[39m
    893:  �[32m+ axios@^1.7.7�[39m
    894:  �[32m+ eslint@^8.57.1�[39m
    895:  �[32m+ eslint-plugin-import@^2.31.0�[39m
    896:  �[32m+ eslint-plugin-jsx-a11y@^6.10.0�[39m
    897:  �[32m+ eslint-plugin-react@^7.37.3�[39m
    898:  �[32m+ eslint-plugin-react-hooks@^4.6.2�[39m
    899:  �[32m+ prettier@^3.3.3�[39m
    900:  �[32m+ react-error-boundary@^4.1.0�[39m
    ...
    
    912:  �[31m- @testing-library/jest-dom@^6.4.2�[39m
    913:  �[31m- axios@^1.7.4�[39m
    914:  �[31m- eslint@^8.48.0�[39m
    915:  �[31m- eslint-plugin-import@^2.29.1�[39m
    916:  �[31m- eslint-plugin-jsx-a11y@^6.8.0�[39m
    917:  �[31m- eslint-plugin-react@^7.34.1�[39m
    918:  �[31m- eslint-plugin-react-hooks@^4.5.0�[39m
    919:  �[31m- prettier@^3.2.5�[39m
    920:  �[31m- react-error-boundary@^4.0.13�[39m
    

    ✨ CI feedback usage guide:

    The CI feedback tool (/checks) automatically triggers when a PR has a failed check.
    The tool analyzes the failed checks and provides several feedbacks:

    • Failed stage
    • Failed test name
    • Failure summary
    • Relevant error logs

    In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR:

    /checks "https://github.com/{repo_name}/actions/runs/{run_number}/job/{job_number}"
    

    where {repo_name} is the name of the repository, {run_number} is the run number of the failed check, and {job_number} is the job number of the failed check.

    Configuration options

    • enable_auto_checks_feedback - if set to true, the tool will automatically provide feedback when a check is failed. Default is true.
    • excluded_checks_list - a list of checks to exclude from the feedback, for example: ["check1", "check2"]. Default is an empty list.
    • enable_help_text - if set to true, the tool will provide a help message with the feedback. Default is true.
    • persistent_comment - if set to true, the tool will overwrite a previous checks comment with the new feedback. Default is true.
    • final_update_message - if persistent_comment is true and updating a previous checks message, the tool will also create a new message: "Persistent checks updated to latest commit". Default is true.

    See more information about the checks tool in the docs.

    Copy link
    Contributor

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
    🧪 PR contains tests
    🔒 Security concerns

    Sensitive information exposure:
    The plugin_report_queries endpoint in api_app/views.py allows querying analyzer/connector reports which may contain sensitive data. While there is basic permission checking, additional controls like TLP-based filtering and data masking should be considered to prevent unauthorized access to sensitive report contents.

    ⚡ Recommended focus areas for review

    Error Handling
    The error handling in _perform_request() method could be improved. The error variable is only used if ignore_404 is False, but errors from other status codes are not properly handled.

    Security Risk
    The plugin_report_queries endpoint exposes potentially sensitive report data. Additional access controls and data filtering should be validated.

    Copy link
    Contributor

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Fix incorrect organization name filtering in Elasticsearch query permissions

    The permission filter in plugin_report_queries uses username instead of organization
    name for the membership filter, which is incorrect and will fail to match
    organization memberships.

    api_app/views.py [1635-1637]

     permission_filter |= QElastic(
    -    "term", membership__organization__name=request.user.username
    +    "terms", membership__organization__name=[org.name for org in request.user.organizations.all()]
     )
    • Apply this suggestion
    Suggestion importance[1-10]: 9

    Why: The suggestion fixes a critical bug in the permission filtering logic that would prevent users from accessing reports from their organizations, as it incorrectly uses username instead of organization names.

    9
    Add request timeout to prevent hanging HTTP requests

    Add request timeout parameter to prevent potential hanging requests. The current
    implementation could wait indefinitely for a response.

    api_app/mixins.py [110]

    -response = requests.get(url, headers=self.headers, **kwargs)
    +response = requests.get(url, headers=self.headers, timeout=30, **kwargs)
    • Apply this suggestion
    Suggestion importance[1-10]: 8

    Why: Adding a timeout parameter is crucial for preventing requests from hanging indefinitely, which could impact system responsiveness and resource utilization. This is especially important for external API calls.

    8
    Validate API credentials before making external requests

    Add validation for the API key before making requests to prevent unnecessary API
    calls with invalid credentials.

    api_app/mixins.py [89-91]

     @property
     def headers(self) -> dict:
    +    if not self._api_key_name:
    +        raise AnalyzerRunException("VirusTotal API key not configured")
         return {"x-apikey": self._api_key_name}
    • Apply this suggestion
    Suggestion importance[1-10]: 8

    Why: Early validation of API credentials prevents unnecessary network calls and provides clearer error messages when credentials are missing or invalid. This improves error handling and system efficiency.

    8
    Add error handling for Elasticsearch connection failures to prevent uncaught exceptions

    Add error handling for the case when Elasticsearch connection fails in the
    plugin_report_queries view. Currently there is no try/catch block around the Search
    execution which could raise connection exceptions.

    api_app/views.py [1674-1677]

    -hits = (
    -    Search(index="plugin-report-*")
    -    .query(QElastic("bool", filter=filter_list))
    -    .execute()
    -)
    +try:
    +    hits = (
    +        Search(index="plugin-report-*")
    +        .query(QElastic("bool", filter=filter_list))
    +        .execute()
    +    )
    +except ConnectionError as e:
    +    logger.error(f"Elasticsearch connection error: {str(e)}")
    +    raise NotImplementedException("Elasticsearch connection failed")
    • Apply this suggestion
    Suggestion importance[1-10]: 8

    Why: The suggestion addresses an important error handling gap that could cause the application to crash when Elasticsearch is unreachable. Adding proper exception handling improves reliability and provides better error feedback.

    8
    Add error handling for base64 encoding of URLs

    Add error handling for base64 encoding to handle potential invalid URL strings that
    could raise exceptions.

    api_app/mixins.py [208-209]

    -url_id = base64.urlsafe_b64encode(observable_name.encode()).decode().strip("=")
    +try:
    +    url_id = base64.urlsafe_b64encode(observable_name.encode()).decode().strip("=")
    +except Exception as e:
    +    raise AnalyzerRunException(f"Invalid URL format: {e}")
    • Apply this suggestion
    Suggestion importance[1-10]: 7

    Why: Adding proper error handling for base64 encoding is important as malformed URLs could cause uncaught exceptions. The suggestion provides clear error messaging and uses the existing exception handling pattern.

    7
    Add validation to ensure file existence before creating rescan job

    The rescan method creates a new job but doesn't validate if the original job's file
    still exists before attempting to use it, which could lead to errors if the file was
    deleted.

    api_app/views.py [571-573]

     if existing_job.is_sample:
    +    if not existing_job.file or not existing_job.file.storage.exists(existing_job.file.name):
    +        raise ValidationError({"detail": "Original job file no longer exists"})
         data["file"] = existing_job.file
         data["file_name"] = existing_job.file_name
    • Apply this suggestion
    Suggestion importance[1-10]: 7

    Why: The suggestion adds important validation to prevent errors when rescanning jobs with missing files, improving error handling and user experience by providing clear feedback.

    7
    Security
    Add memory and file size checks when processing potentially malicious zip files

    Add error handling around the zipfile.ZipFile() call to catch potential memory
    issues with malformed/malicious zip files. Consider adding a maximum file size
    check.

    api_app/analyzers_manager/file_analyzers/doc_info.py [276-279]

     try:
    +    if os.path.getsize(self.filepath) > settings.MAX_FILE_SIZE:
    +        raise ValueError("File too large")
         document = zipfile.ZipFile(self.filepath)
    -except zipfile.BadZipFile as e:  # check if docx document
    -    error_message = f"job_id {self.job_id} docx bad zip file: {e}"
    +except (zipfile.BadZipFile, ValueError, MemoryError) as e:
    +    error_message = f"job_id {self.job_id} docx processing failed: {e}"
    • Apply this suggestion
    Suggestion importance[1-10]: 9

    Why: Critical security enhancement that prevents potential DoS attacks through malformed zip files. The suggestion adds important memory and file size validation checks.

    9
    Prevent XML External Entity (XXE) attacks by properly configuring XML parser security options

    Add validation for the XML content before parsing to prevent XXE attacks. Use
    defusedxml's parse() with forbid_dtd=True.

    api_app/analyzers_manager/file_analyzers/doc_info.py [284-290]

     dxml = document.read("docProps/app.xml")
    +parser = parseString(dxml, forbid_dtd=True)
     pages_count = int(
    -    parseString(dxml)
    +    parser
         .getElementsByTagName("Pages")[0]
         .childNodes[0]
         .nodeValue
     )
    • Apply this suggestion
    Suggestion importance[1-10]: 8

    Why: Important security improvement that prevents XXE attacks by adding proper XML parser configuration. The suggestion enhances the security of XML processing.

    8
    Validate and sanitize extracted URLs to prevent injection attacks

    Add input validation for the URL extraction to prevent potential command injection
    or path traversal attacks. Consider using urllib.parse to validate URLs.

    api_app/analyzers_manager/file_analyzers/doc_info.py [324-328]

    -if relationship.attrib["Type"] == XML_H_SCHEMA and any(
    -    domain in relationship.attrib["Target"]
    -    for domain in SCHEMA_DOMAINS
    -):
    -    urls.append(relationship.attrib["Target"])
    +from urllib.parse import urlparse
    +target = relationship.attrib.get("Target", "")
    +if relationship.attrib.get("Type") == XML_H_SCHEMA:
    +    parsed = urlparse(target)
    +    if any(domain in parsed.netloc for domain in SCHEMA_DOMAINS):
    +        urls.append(target)
    • Apply this suggestion
    Suggestion importance[1-10]: 7

    Why: Valuable security enhancement that adds proper URL validation and sanitization to prevent potential injection attacks through malicious URLs.

    7

    @gitworkflows
    Copy link
    Contributor Author

    gitworkflows commented Dec 26, 2024

    Snyk checks have failed. 18 issues have been found so far.

    Icon Severity Issues
    Critical 1
    High 17
    Medium 0
    Low 0

    security/snyk check is complete. 18 issues have been found. (View Details)

    gitworkflows and others added 4 commits December 27, 2024 05:34
    * Update serializers.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update urls.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update views.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update urls.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update views.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update urls.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update views.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update serializers.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update urls.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Update views.py
    
    Signed-off-by: gitworkflows <[email protected]>
    
    * Refactor Plugin Config
    
    * Refactor Plugin Config
    
    ---------
    
    Signed-off-by: gitworkflows <[email protected]>
    Signed-off-by: gitworkflows <[email protected]>
    Signed-off-by: gitworkflows <[email protected]>
    Bumps [eslint-plugin-react](https://github.com/jsx-eslint/eslint-plugin-react) from 7.37.1 to 7.37.3.
    - [Release notes](https://github.com/jsx-eslint/eslint-plugin-react/releases)
    - [Changelog](https://github.com/jsx-eslint/eslint-plugin-react/blob/v7.37.3/CHANGELOG.md)
    - [Commits](jsx-eslint/eslint-plugin-react@v7.37.1...v7.37.3)
    
    ---
    updated-dependencies:
    - dependency-name: eslint-plugin-react
      dependency-type: direct:development
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    2 participants