feat: enhance KQL parser with escaped character support and performance tests

[laojianzi]

Close #3

Summary by Sourcery

Enhance the KQL parser to support escaped characters, improving query flexibility and robustness. Update documentation with detailed usage and examples, and introduce comprehensive tests and benchmarks to ensure performance and reliability.

New Features:

• Add support for escaped characters in the KQL parser, allowing for more flexible query expressions.

Enhancements:

• Improve the lexer and parser to handle escaped characters and wildcards more efficiently.
• Refactor the lexer to include new functions for handling tokens and escape sequences, improving code readability and maintainability.

Documentation:

• Update README.md with detailed usage instructions, installation steps, and examples for the KQL parser.
• Enhance package documentation to include more comprehensive information about features, performance, and error handling.

Tests:

• Introduce extensive test cases for escaped characters, operators, wildcards, and complex queries to ensure robust parsing.
• Add performance benchmarks to evaluate the efficiency of the parser and lexer under various conditions.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR significantly improves the KQL parser's handling of escape sequences and adds comprehensive testing infrastructure. The changes include a new escape handling system, extensive test coverage, benchmarking capabilities, and improved documentation.

Class diagram for updated Literal class

classDiagram
    class Literal {
        int pos
        int end
        token.Kind Kind
        string Value
        bool WithDoubleQuote
        int[] escapeIndexes
        +NewLiteral(int pos, int end, token.Kind kind, string value, int[] escapeIndexes)
        +String() string
    }
    note for Literal "Added escapeIndexes attribute to handle escape sequences"

Loading

Class diagram for new CharProcResult struct

classDiagram
    class CharProcResult {
        int Position
        bool IsEscaped
        int[] EscapeIndexes
        +NewCharProcResult(int position, bool isEscaped, int[] escapeIndexes)
        +String() string
    }
    note for CharProcResult "New struct to handle character processing results in token lexing"

Loading

Class diagram for updated defaultLexer class

classDiagram
    class defaultLexer {
        rune[] Value
        Token Token
        int pos
        token.Kind lastTokenKind
        bool dotIdent
        +newLexer(string input) *defaultLexer
        +nextToken() error
        +consumeToken() error
        +consumeFieldToken() error
        +consumeEscapedToken(token.Kind kind, rune endChar) (int, []int, error)
        +consumeIdent() error
        +consumeString() error
        +consumeNumber() error
        +consumeOperator() error
        +consumeParen() error
        +skipSpaces()
        +skipN(int n)
        +peek(int i) rune
        +peekOk(int i) bool
        +slice(int start, int end) string
        +eof() bool
        +shouldBreak(int i, bool isString, bool withEscape, rune endChar) bool
        +collectNextToken(int start) string
        +processNonEscaped(int i, token.Kind kind) bool
        +handleNonEscaped(int pos, token.Kind k, *bytes.Buffer buf, []int indexes) CharProcResult
        +handleEscaped(int pos, token.Kind k, *bytes.Buffer buf, []int indexes) (CharProcResult, error)
        +handleBackslash(int pos, *bytes.Buffer buf, bool isString, []int indexes) (int, []int)
        +handleEscapeSequence(int pos, token.Kind k) (bool, error)
    }
    note for defaultLexer "Added methods to handle escape sequences and token processing"

Loading

File-Level Changes

Change	Details	Files
Implemented a new escape sequence handling system	Added new CharProcResult struct to manage escape sequence processing Created dedicated escape handling functions for different token types Implemented validation for escape sequences Added support for escaping special characters, keywords, and operators	parser/escape.go parser/lexer.go parser/parser.go token/token.go
Added comprehensive testing infrastructure	Added benchmark tests for parser performance Implemented fuzzing tests for parser robustness Created test cases for escaped characters and edge cases Added GitHub workflow for performance benchmarking	parser/bench_test.go parser/fuzz_test.go parser/parser_test.go .github/workflows/benchmark.yml
Enhanced documentation and project structure	Updated README with comprehensive feature list and examples Improved package documentation with detailed usage examples Added Dependabot configuration for automated dependency updates Updated code organization and import ordering	README.md doc.go .github/dependabot.yml

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#3	Support escaping operator characters (<, >, :)	✅
#3	Support escaping parentheses characters ( and )	✅
#3	Support escaping keyword characters (and, or, not)	✅
#3	Support escaping wildcard and backslash characters (* and ) in strings	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @laojianzi - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟡 Complexity: 1 issue found
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[deepsource-io]

Here's the code health analysis summary for commits ce58a86..874520b. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
Go	✅ Success		View Check ↗
Test coverage	✅ Success	❗ 12 occurences introduced 🎯 11 occurences resolved	View Check ↗

Code Coverage Report

Metric	Aggregate	Go
Composite Coverage	85.3% (up 2.1% from main)	85.3% (up 2.1% from main)
Line Coverage	85.3% (up 2.1% from main)	85.3% (up 2.1% from main)
New Composite Coverage	79.2%	79.2%
New Line Coverage	79.2%	79.2%

---

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

[llamapreview]

Auto Pull Request Review from LlamaPReview

Large PR Notification

Dear contributor,

Thank you for your substantial contribution to this project. LlamaPReview has detected that this Pull Request contains a large volume of changes, which exceeds our current processing capacity.

Details:

• PR and related contents total size: Approximately 80,759 characters
• Current limit: 50,000 characters

Next steps:

1. Consider breaking this PR into smaller, more focused changes if possible.
2. For manual review, please reach out to your team members or maintainers.

We appreciate your understanding and commitment to improving this project. Your contributions are valuable, and we want to ensure they receive the attention they deserve.

LlamaPReview is continuously evolving to better serve the community. Share your thoughts on handling large PRs in our GitHub Discussions - your feedback helps us improve and expand our capabilities.

If you have any questions or need assistance, our community and support team are here to help.

Best regards,
LlamaPReview Team

[laojianzi]

@sourcery-ai

[laojianzi]

@sourcery-ai summary

[sourcery-ai]

I'm sorry, I don't understand the command @sourcery-ai summary.

Please use @sourcery-ai review to request a review.