Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update -tc flag to filter fields within the classification section #3606

Merged
merged 4 commits into from
Apr 26, 2023
Merged

Update -tc flag to filter fields within the classification section #3606

merged 4 commits into from
Apr 26, 2023

Conversation

iamargus95
Copy link
Contributor

@iamargus95 iamargus95 commented Apr 25, 2023 •
edited
Loading

Proposed changes

  • Update -tc to support selecting key/values under classification section from a template.
  • Fix existing implementation of filtering Metadata fields.
    • If the field names are hyphenated, the evaluation on them failed.

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

This was linked to issues Apr 25, 2023
template-condition update to support selecting key/values under classification #3575
Closed
template-condition flag fails to evaluate hyphenated fields from metadata section #3605
Closed
@iamargus95 iamargus95 marked this pull request as ready for review April 25, 2023 11:20
@ShubhamRasal ShubhamRasal requested a review from ehsandeep April 25, 2023 11:59
@ehsandeep ehsandeep requested a review from Mzack9999 April 25, 2023 22:29
@ehsandeep ehsandeep added the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label Apr 25, 2023
Mzack9999
Mzack9999 approved these changes Apr 26, 2023
View reviewed changes
Copy link
Member

@Mzack9999 Mzack9999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - Note: unfortunately lots of metadata is forcefully normalized (eg. lower cased) during marshaling/unmarshaling, hence the filtering needs to be adapted accordingly, for example -tc "contains(cve_id,'cve-2000-0114')" will work but -tc "contains(cve_id,'CVE-2000-0114')" will not.

@ehsandeep ehsandeep merged commit 78f1ade into projectdiscovery:dev Apr 26, 2023
@ehsandeep ehsandeep removed the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label Apr 26, 2023
@ehsandeep
Copy link
Member

Thank you @iamargus95 for adding this support.

@iamargus95 iamargus95 deleted the issue-3575-update-template-condition-flag branch April 27, 2023 06:35
RamanaReddy0M pushed a commit that referenced this pull request May 3, 2023
@iamargus95 @RamanaReddy0M
Update -tc flag to filter fields within the classification section (#…
8b0ad48 
…3606)

* Add fields from Classification section in a template to the -tc flag expression evaluation

Signed-off-by: iamargus95 <[email protected]>

* Add tests for filtering Classification section using -tc flag

Signed-off-by: iamargus95 <[email protected]>

* Fix hyphenated Metadata keys beings added to parameters

Signed-off-by: iamargus95 <[email protected]>

* Add tests to the fix for hyphenated fields encountered in Metadata section

Signed-off-by: iamargus95 <[email protected]>

---------

Signed-off-by: iamargus95 <[email protected]>
ehsandeep added a commit that referenced this pull request May 9, 2023
@RamanaReddy0M @ShubhamRasal
@iamargus95 @dogancanbakir @Mzack9999 @ehsandeep @tarunKoyalwar @dependabot @lu4nx
Add utility to write max-requests to templates (#3607)
5937618 
* Add utility to write max-requests to templates

* fix lint error

* fix max-request update edge case

* fix convert max-request: 1 => max-request: 1

* WIP, most of the code is commented

* Refactor the find and replace  logic

* Skip if template has the max-requests, do not overwrite

- return errors
- add warnings

* Fix the wrong index calculation

- Refactor the getInfoBlock function to not compile regex everytime

* Update -tc flag to filter fields within the classification section (#3606)

* Add fields from Classification section in a template to the -tc flag expression evaluation

Signed-off-by: iamargus95 <[email protected]>

* Add tests for filtering Classification section using -tc flag

Signed-off-by: iamargus95 <[email protected]>

* Fix hyphenated Metadata keys beings added to parameters

Signed-off-by: iamargus95 <[email protected]>

* Add tests to the fix for hyphenated fields encountered in Metadata section

Signed-off-by: iamargus95 <[email protected]>

---------

Signed-off-by: iamargus95 <[email protected]>

* Check severity att while validating (#3540)

* Make severity attribute required

* Update test err msg

* minor

* Do not strict check serverity

* Fix failing test

* Don't print warning in workflow loader

- workflow loader that contains tags load all the template and parse it
- i.e it iw printing warning recursively, ignore as the templates
  already getting valiated

* Fix error typo

* Resolve comments

- split the function into two diff

---------

Co-authored-by: Mzack9999 <[email protected]>
Co-authored-by: Sandeep Singh <[email protected]>
Co-authored-by: shubhamrasal <[email protected]>

* tlsx dep update (#3620)

* updated interactsh version (#3621)

* updated interactsh version

* workflow update

* aws signer: fix missing x-content-sha256 header (#3601)

* fix missing x-content-sha256 header

* fix variable priority in self-contained templates

* remove debug statement

* adds generic raw request parser for self-contained req

* more integration tests

* bug fix: 10x faster race requests

* fix failing integration test

* chore(deps): bump github.com/xanzy/go-gitlab in /v2 (#3624)

Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.82.0 to 0.83.0.
- [Release notes](https://github.com/xanzy/go-gitlab/releases)
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](xanzy/go-gitlab@v0.82.0...v0.83.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/miekg/dns from 1.1.53 to 1.1.54 in /v2 (#3625)

Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.53 to 1.1.54.
- [Release notes](https://github.com/miekg/dns/releases)
- [Changelog](https://github.com/miekg/dns/blob/master/Makefile.release)
- [Commits](miekg/dns@v1.1.53...v1.1.54)

---
updated-dependencies:
- dependency-name: github.com/miekg/dns
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /v2 (#3626)

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.61 to 1.11.64.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.61...feature/s3/manager/v1.11.64)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix check for OS made in MustDisableSandbox() (#3631)

Signed-off-by: iamargus95 <[email protected]>

* Fix wrong template loading in dev branch (#3629)

* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded

* enable color in windows (#3634)

* enable color in windows

* fixed win workflow

* typo update

* tlsx dep update (#3633)

* tlsx dep update

* upgrde httpx => 1.3.0

* Fix check for OS made in MustDisableSandbox() (#3631)

Signed-off-by: iamargus95 <[email protected]>

* Fix wrong template loading in dev branch (#3629)

* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded

* enable color in windows (#3634)

* enable color in windows

* fixed win workflow

* typo update

---------

Signed-off-by: iamargus95 <[email protected]>
Co-authored-by: Ramana Reddy <[email protected]>
Co-authored-by: Suraj Kamath <[email protected]>
Co-authored-by: Shubham Rasal <[email protected]>

* Expose DNS fields for matchers and extractors (#3613)

* Extend dns extractor to dns answer records

* add test template

* Ignore error for dns variables are not found

* Add all the records of answer section

* Fixed the wrong typecasting

* Issue 3564 var override (#3599)

* Check if the variables are override by other means

- you can override the template variable value using command line flags

* Update lazy eval logic

- previously, we were checking any function/expression in variable
- now, update the logic, lazy eval only if variable contains any
  protocol variable(global)

* add integration tests

* Add test to check the dsl function working in variable

* gather all generate variables logic in utils

* go mod update

* Refactor the generate variables function

* go mod update+ fix typo

---------

Co-authored-by: Sandeep Singh <[email protected]>
Co-authored-by: sandeep <[email protected]>
Co-authored-by: Tarun Koyalwar <[email protected]>

* update rod to v0.112.9 #3552 (#3637)

* update rod to v0.112.9

* removed unused reflection

---------

Co-authored-by: Mzack9999 <[email protected]>
Co-authored-by: Sandeep Singh <[email protected]>
Co-authored-by: sandeep <[email protected]>

* change max-requests label to max-request

---------

Signed-off-by: iamargus95 <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: shubhamrasal <[email protected]>
Co-authored-by: Suraj Kamath <[email protected]>
Co-authored-by: Dogan Can Bakir <[email protected]>
Co-authored-by: Mzack9999 <[email protected]>
Co-authored-by: Sandeep Singh <[email protected]>
Co-authored-by: Tarun Koyalwar <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shubham Rasal <[email protected]>
Co-authored-by: sandeep <[email protected]>
Co-authored-by: Tarun Koyalwar <[email protected]>
Co-authored-by: lu4nx <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mzack9999 Mzack9999 Mzack9999 approved these changes

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees

@iamargus95 iamargus95

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@iamargus95 @ehsandeep @Mzack9999