Skip to content

Accept Reborn live QA success wording variants #5436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serrrfirat merged 1 commit into from
Jun 30, 2026
Merged

Accept Reborn live QA success wording variants #5436

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions scripts/reborn_webui_v2_live_qa/run_live_qa.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1185,7 +1185,7 @@ async def case_qa_3b_endpoint_status_live_chat(ctx: LiveQaContext) -> ProbeResul
case_name="qa_3b_endpoint_status_live_chat",
prompt=_qa_sheet_prompt("qa_3b_endpoint_status_live_chat"),
marker=None,
required_text=["status"],
required_text=["status|http|200|up|running|responded"],

@coderabbitai coderabbitai Bot Jun 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

These gates became single-token OR matches.

_required_text_matches() ANDs list elements but ORs the | terms inside each element. With one pipe-delimited string here, the case passes on any one token: bare 200/up, bare created, or bare discussion/id=. That weakens the live QA gate enough to greenlight unrelated replies.

Suggested fix 
-        required_text=["status|http|200|up|running|responded"],
+        required_text=["status|http|responded", "200", "up|running"],
...
-        required_text=["routine|trigger|automation|cron|schedule|created"],
+        required_text=["routine|trigger|automation|cron|schedule", "created"],
...
-        required_text=["news.ycombinator.com|hacker news|hn|discussion|id="],
+        required_text=["news.ycombinator.com|hacker news|hn", "discussion|item", "id="],

Also applies to: 2771-2771, 3314-3314

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/reborn_webui_v2_live_qa/run_live_qa.py` at line 1188, The live QA
gate in _required_text_matches is too loose because required_text is a single
pipe-delimited string, so any one token can satisfy the check. Update the
affected required_text entries in run_live_qa.py to use separate list items for
each required signal, and tighten the gate so it requires all intended
conditions rather than OR-ing unrelated tokens; verify the same pattern in the
other highlighted call sites as well.

extra_details={"endpoint_url": url, "expected_status_code": live_status},
)

Expand Down Expand Up @@ -2440,7 +2440,7 @@ async def case_qa_6c_gmail_to_sheet_live_chat(ctx: LiveQaContext) -> ProbeResult
ctx,
case_name="qa_6c_gmail_to_sheet_live_chat",
marker=None,
required_text=["ABC", "spreadsheet"],
required_text=["ABC|sheet|spreadsheet", "email|row|near.ai|near ai"],
extensions=[
{
"package_id": "gmail",
Expand Down Expand Up @@ -2768,7 +2768,7 @@ async def case_qa_4d_github_release_slack_routine(ctx: LiveQaContext) -> ProbeRe
case_name="qa_4d_github_release_slack_routine",
routine_name=routine_name,
marker=None,
required_text=["routine"],
required_text=["routine|trigger|automation|cron|schedule|created"],
prompt=_qa_sheet_prompt("qa_4d_github_release_slack_routine"),
)

Expand Down Expand Up @@ -3311,7 +3311,7 @@ async def case_qa_8b_hn_keyword_live_chat(ctx: LiveQaContext) -> ProbeResult:
case_name="qa_8b_hn_keyword_live_chat",
prompt=_qa_sheet_prompt("qa_8b_hn_keyword_live_chat"),
marker=None,
required_text=["news.ycombinator.com"],
required_text=["news.ycombinator.com|hacker news|hn|discussion|id="],
timeout=240.0,
)

Expand Down
31 changes: 29 additions & 2 deletions scripts/reborn_webui_v2_live_qa/test_run_live_qa.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,30 @@ def test_required_text_accepts_explicit_alternatives(self):
["trigger|routine|automation|cron|schedule|fires|watches", "bug"],
)
)
self.assertTrue(
run_live_qa._required_text_matches(
"https://near.ai responded with HTTP 200 - the endpoint is up and running fine.",
["status|http|200|up|running|responded"],
)
)
self.assertTrue(
run_live_qa._required_text_matches(
"Trigger created. Schedule: every 5 minutes. Action: fetch latest releases.",
["routine|trigger|automation|cron|schedule|created"],
)
)
self.assertTrue(
run_live_qa._required_text_matches(
"The email from firat.sertgoz@near.ai is already in the sheet.",
["ABC|sheet|spreadsheet", "email|row|near.ai|near ai"],
)
)
self.assertTrue(
run_live_qa._required_text_matches(
'Discussion thread "vibe coded eh" (id=47005839) mentions NEAR AI.',
["news.ycombinator.com|hacker news|hn|discussion|id="],
)
)

def test_slack_delivery_target_dm_detection(self):
self.assertTrue(run_live_qa._slack_delivery_target_is_dm("D12345"))
Expand Down Expand Up @@ -825,7 +849,10 @@ async def fake_live_chat_case(_ctx, **kwargs):
)

self.assertTrue(result.success)
self.assertEqual(captured["required_text"], ["news.ycombinator.com"])
self.assertEqual(
captured["required_text"],
["news.ycombinator.com|hacker news|hn|discussion|id="],
)

def test_live_google_side_effect_cases_install_required_extensions(self):
captured: dict[str, dict[str, object]] = {}
Expand Down Expand Up @@ -939,7 +966,7 @@ async def fake_google_sheet_contains_marker(**_kwargs):
)
self.assertEqual(
captured["qa_6c_gmail_to_sheet_live_chat"]["required_text"],
["ABC", "spreadsheet"],
["ABC|sheet|spreadsheet", "email|row|near.ai|near ai"],
)
self.assertTrue(
extensions_by_case["qa_2f_calendar_prep_email_delivery"]["google-docs"].get(
Expand Down
Loading