fix: prevent lexer byte-offset drift on invalid UTF-8 input

[zainnadeem786]

Summary

Fix a slice-bounds panic caused by invalid UTF-8 input drifting the lexer's byte cursor away from the original expression.

This issue was originally identified while investigating a Nuclei crash reported in projectdiscovery/nuclei#7462.

Root Cause

lexerStream maintains two positions while parsing expressions:

• position (rune index)
• strPosition (byte offset into the original source string)

When invalid UTF-8 bytes are encountered, Go's string iteration decodes them as utf8.RuneError while consuming a single byte from the source string.

However, the lexer advanced strPosition using:

utf8.RuneLen(character)

For utf8.RuneError, utf8.RuneLen returns 3, causing the byte cursor to drift ahead of the actual source position.

A later token could then enter the reuse-string fast path in readUntilFalse() and attempt to slice beyond the end of sourceString, resulting in:

panic: runtime error: slice bounds out of range

Fix

Use the actual decoded byte width when handling utf8.RuneError during both forward and backward cursor movement.

For valid UTF-8 input, the existing fast path and behavior remain unchanged.

Tests

Added regression coverage for:

• Multiple invalid UTF-8 bytes (reproduces nuclei#7462)
• Single invalid UTF-8 byte
• Valid multibyte UTF-8 characters
• Encoded U+FFFD characters
• Truncated UTF-8 sequences

Verified that expressions continue to tokenize correctly and no panic occurs.

Validation

go test -run '^TestUTF8Parsing$' -count=1
go test -count=1
go vet ./...

All tests pass.

Risk Assessment

Low risk.

The change is isolated to lexer cursor accounting for invalid UTF-8 input.

• Valid UTF-8 behavior is unchanged.
• Tokenization behavior remains unchanged.
• Existing allocations and parsing paths remain unchanged.
• Malformed input continues to decode as replacement runes but no longer corrupts byte-offset tracking.

References

• Fixes the root cause behind [BUG] govaluate: slice bounds out of range panic in readUntilFalse on invalid UTF-8 input nuclei#7462