chore: Simple log filter script

[paultranvan]

Summary by CodeRabbit

• New Features
  • Added a command-line log-filtering utility to extract NDJSON records whose embedded timestamps fall within a specified inclusive range. Supports UTC or local timezone resolution, multiple input/output options, optional retention of unparseable records, and an option to emit processing statistics.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a new CLI Python script that filters NDJSON log records by ISO-like timestamps extracted from each record's text field, with timezone-aware parsing, start/end range validation, optional retention of unparseable lines, and optional processing statistics.

Changes

Cohort / File(s)	Summary
New NDJSON log filter script openrag/scripts/filter-logs.py	New script implementing CLI parsing (--start, --end, --tz, --keep-invalid, --stats), compiled regex TEXT_TS_RE, parse_text_timestamp(text, assume_tz), parse_cli_datetime(s, assume_tz), main(), timezone resolution (UTC or system local), NDJSON line-by-line JSON parsing, timestamp extraction, inclusive range filtering, optional writing of invalid lines, and optional stderr stats.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as Argument Parser
    participant TZ as Timezone Resolver
    participant Reader as NDJSON Reader
    participant Parser as Text Timestamp Parser
    participant Filter as Range Filter
    participant Writer as Output Writer
    participant Stats as Stats Reporter

    User->>CLI: invoke with args (--start,--end,--tz,...)
    CLI->>TZ: resolve timezone (UTC or LOCAL)
    CLI->>Reader: open input file and pass params
    loop per line
        Reader->>Parser: parse JSON, extract `text`, match `TEXT_TS_RE`
        Parser->>Filter: return datetime or None
        Filter->>Writer: if in range -> write line
        alt invalid and --keep-invalid
            Filter->>Writer: write line
        end
    end
    alt --stats
        Writer->>Stats: provide counts
        Stats->>User: emit stats to stderr
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I sniff the timestamps, one by one,
I hop through lines until the job is done.
UTC or local, I sort with glee,
Keep the good logs, or keep all—whee!
A tiny script, from me to thee.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change—a new log filtering script—and is concise and clear.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

🧹 Recent nitpick comments

openrag/scripts/filter-logs.py (1)

120-127: Consider moving the sys import to the top of the file.

The lazy import works but is unconventional. Since sys is a standard library module with negligible import cost, placing it with other imports at the top improves readability.

Suggested change

 import argparse
 import json
 import re
+import sys
 from datetime import datetime, timezone
 from typing import Optional

     if args.stats:
-        import sys
         print(f"Read lines: {in_count}", file=sys.stderr)

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c260d30 and 241eada.

📒 Files selected for processing (1)

• openrag/scripts/filter-logs.py

🧰 Additional context used

🪛 Ruff (0.14.11)

openrag/scripts/filter-logs.py

50-50: Avoid specifying long messages outside the exception class

(TRY003)

---

76-76: Avoid specifying long messages outside the exception class

(TRY003)

---

82-82: Avoid specifying long messages outside the exception class

(TRY003)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: api-tests
• GitHub Check: index-backup-restore

🔇 Additional comments (6)

openrag/scripts/filter-logs.py (6)

1-12: LGTM!

The imports are minimal and appropriate. The regex pattern correctly captures the expected timestamp format with optional microseconds using named groups for clarity.

---

14-32: LGTM!

The timestamp parsing logic is correct. The microsecond normalization by right-padding with zeros is the appropriate interpretation (e.g., .538 = 538000 microseconds = 0.538 seconds).

---

34-50: LGTM!

The CLI datetime parsing is flexible and handles common ISO-8601 variants appropriately.

---

52-68: LGTM!

The argument parsing is well-structured with clear help text and sensible defaults.

---

70-82: LGTM!

The timezone handling is appropriate for a simple script without external dependencies. The fallback to timezone.utc when tzinfo is None is a good defensive measure.

---

84-118: LGTM!

The line-by-line processing logic is correct:

• Preserves original line formatting when writing output
• Properly handles JSON decode errors and unparseable timestamps
• Inclusive range filtering matches the documented behavior

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}