Model Availability Polling and ELO-Based Dynamic Routing by google-labs-jules[bot] · Pull Request #123 · timerloggedout-spec/termux-monorepo

google-labs-jules · 2026-08-09T21:52:54Z

Implemented model availability polling for OpenRouter free models and transitioned to our internal historic performance index (ELO ≈ 3L0) to route triage, review, and invoke tasks dynamically. Verified using a comprehensive set of unit tests (all passed successfully).

Fixes #122

PR created automatically by Jules for task 3359099756114716988 started by @timerloggedout-spec

…outing (#119) - Replaced rigid bash picking logic inside GHA model router with `scripts/model_router.py`. - Implemented real-time OpenRouter `:free` model availability polling via live API endpoint. - Introduced custom performance index (ELO ≈ 3L0) in `docs/schemas/model-success-matrix.yaml` to phase out reliance on public leaderboards. - Maintained 100% GHA cache budget compatibility and added comprehensive pytest unit tests under `tests/test_model_router.py`.

google-labs-jules · 2026-08-09T21:52:55Z

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

For security, I will only act on instructions from the user who triggered this task.

vercel · 2026-08-09T21:52:56Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
termux-monorepo	Ready	Preview, v0	Aug 10, 2026 2:57am

blocksorg · 2026-08-09T21:52:57Z

Mention Blocks like a regular teammate with your question or request:

@blocks review this pull request
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

github-actions · 2026-08-09T21:56:21Z

head_sha: d82fb4f
ready: true
autofix_requested: false
timed_out: false

Peer review gate (ready for second-pass agents)

External reviewers polled: CodeRabbit, Devin, Aikido, Sentry, Copilot.
Autofix (if any) was requested in a separate comment on this SHA.

Peer activity (truncated):

review @devin-ai-integration[bot] state=COMMENTED sha=d82fb4f

Downstream: gemini-after-peers. Jules: agent-review-auto-jules.

github-actions · 2026-08-09T21:56:25Z

@jules Auto-resolve (GHA agent-review-auto-jules) — do not wait for a human ping.
Bot feedback from devin-ai-integration[bot] on PR #123 (branch jules-3359099756114716988-8a9652be).

Feedback excerpt

**Devin Review** found 8 potential issues.

<!-- devin-review-badge-begin -->
<a href="https://app.devin.ai/review/timerloggedout-spec/termux-monorepo/pull/123" target="_blank">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
    <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review">
  </picture>
</a>
<!-- devin-review-badge-end -->

Instructions

Address all open review threads on this PR (CodeRabbit, Devin, Copilot, etc.).
Prefer minimal diffs; preserve Sentinel 0o600/0o700 if those files are touched.
Push commits to branch jules-3359099756114716988-8a9652be. Do not retarget away from the PR base without cause.
If conflicts with base exist, resolve them.
Skip pure nits only if they conflict with security/gates; otherwise apply autofixes.
Agent: Grok orchestration · Profile: https://x.com/grok

…119) - Replaced rigid bash picking logic inside GHA model router with `scripts/model_router.py`. - Implemented real-time OpenRouter `:free` model availability polling via live API endpoint. - Introduced custom performance index (ELO ≈ 3L0) in `docs/schemas/model-success-matrix.yaml` to phase out reliance on public leaderboards. - Catch Linear 'usage limit exceeded' / 'USAGE_LIMIT_EXCEEDED' GraphQL errors gracefully in `agent-feedback-linear-sync.yml` to prevent CI failure when free tiers are exhausted. - Maintained 100% GHA cache budget compatibility and added comprehensive pytest unit tests under `tests/test_model_router.py`.

github-actions · 2026-08-09T22:02:36Z

head_sha: 3bedc85
ready: true
autofix_requested: false
timed_out: false

Peer review gate (ready for second-pass agents)

External reviewers polled: CodeRabbit, Devin, Aikido, Sentry, Copilot.
Autofix (if any) was requested in a separate comment on this SHA.

Peer activity (truncated):

review @devin-ai-integration[bot] state=COMMENTED sha=3bedc85

Downstream: gemini-after-peers. Jules: agent-review-auto-jules.

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…ching, robust fallback, security checks, and graceful Linear sync limit handling (#119) - Replaced rigid GHA bash router with `scripts/model_router.py`. - Added 1-hour temporal file caching for OpenRouter `:free` model list polling to optimize polling timing/intervals. - Guarded polling behind `has_openrouter` to completely eliminate redundant network latency on restricted runners. - Asserted `:free` suffix check on all selected OpenRouter models for robust safety enforcement. - Wrapped `main()` in a top-level try-except for seamless graceful degradation inside actions. - Moved comment reaction inside try-except for Linear issue synchronization in `agent-feedback-linear-sync.yml` to prevent fake success reactions on limit errors. - Added comprehensive unit tests in `tests/test_model_router.py` (all passed).

timerloggedout-spec · 2026-08-09T22:26:01Z

@jules @google-labs-jules @coderabbitai

Evaluate and fix everything necessary for performant runs executed in this PR; including:
#123 (review)
#123 (review)

Make sure everything is either maintained or upgraded.

- Replaced rigid GHA bash router with `scripts/model_router.py`. - Added 1-hour temporal file caching for OpenRouter `:free` model list polling to optimize polling timing/intervals. - Guarded polling behind `has_openrouter` to completely eliminate redundant network latency on restricted runners. - Asserted `:free` suffix check on all selected OpenRouter models for robust safety enforcement. - Wrapped `main()` in a top-level try-except for seamless graceful degradation inside actions. - Moved comment reaction inside try-except for Linear issue synchronization in `agent-feedback-linear-sync.yml` to prevent fake success reactions on limit errors. - Added comprehensive unit tests in `tests/test_model_router.py` (all passed). - Updated `docs/proposals/active/rate-limit-rotation/ITEMS.md` tracking row. Implements: RL-17 Signed-off-by: Jules <jules@grok.x.ai>

github-actions · 2026-08-09T22:34:09Z

head_sha: 2be3c17
ready: true
autofix_requested: false
timed_out: false

Peer review gate (ready for second-pass agents)

External reviewers polled: CodeRabbit, Devin, Aikido, Sentry, Copilot.
Autofix (if any) was requested in a separate comment on this SHA.

Peer activity (truncated):

review @devin-ai-integration[bot] state=COMMENTED sha=2be3c17 | review @devin-ai-integration[bot] state=COMMENTED sha=2be3c17 | review @devin-ai-integration[bot] state=COMMENTED sha=2be3c17

Downstream: gemini-after-peers. Jules: agent-review-auto-jules.

github-actions · 2026-08-09T22:34:11Z

@jules Auto-resolve (GHA agent-review-auto-jules) — do not wait for a human ping.
Bot feedback from devin-ai-integration[bot] on PR #123 (branch jules-3359099756114716988-8a9652be).
File: scripts/model_router.py

Feedback excerpt

✅ **Resolved**: The current change adds RL-17 to the active proposal ITEMS.md, and the current head commit message contains `Implements: RL-17`; the mandatory tracking/citation requirement is now met.

Instructions

Address all open review threads on this PR (CodeRabbit, Devin, Copilot, etc.).
Prefer minimal diffs; preserve Sentinel 0o600/0o700 if those files are touched.
Push commits to branch jules-3359099756114716988-8a9652be. Do not retarget away from the PR base without cause.
If conflicts with base exist, resolve them.
Skip pure nits only if they conflict with security/gates; otherwise apply autofixes.
Agent: Grok orchestration · Profile: https://x.com/grok

github-actions · 2026-08-09T22:34:29Z

🔀 OpenRouter review (`qwen/qwen3-coder:free`)

⚠️ openrouter returned no content. This model is unavailable for free. The paid version is available now - use this slug instead: qwen/qwen3-coder

Peer router: Omni ↔ OpenRouter by desired model; Gemini residual. role=review

- Replaced rigid GHA bash router with `scripts/model_router.py`. - Added 1-hour temporal file caching for OpenRouter `:free` model list polling to optimize polling timing/intervals. - Guarded polling behind `has_openrouter` to completely eliminate redundant network latency on restricted runners. - Asserted `:free` suffix check on all selected OpenRouter models for robust safety enforcement. - Wrapped `main()` in a top-level try-except for seamless graceful degradation inside actions. - Moved comment reaction inside try-except for Linear issue synchronization in `agent-feedback-linear-sync.yml` to prevent fake success reactions on limit errors. - Added comprehensive unit tests in `tests/test_model_router.py` (all passed). - Updated `docs/proposals/active/rate-limit-rotation/ITEMS.md` tracking row. Implements: RL-17 Signed-off-by: Jules <jules@grok.x.ai>

timerloggedout-spec

Approved as OPERATOR. Model availability polling + ELO dynamic routing advances RL-05 / #122. Devin findings noted for follow-up; unit tests reported green. Merge when checks clear.

timerloggedout-spec · 2026-08-10T03:03:10Z

OPERATOR merge complete.

#123 (Model Availability Polling + ELO Dynamic Routing) merged to master via Connectors.

SHA: b031cbf1fc71e07afc20049a4bda62ba3d7cb0bc
Fixes Model Availability Polling #122
Implements: RL-17
Advances rate-limit-rotation RL-05

All open review threads resolved prior to merge. Follow-up: rebase #104 / #105 / #114 onto new master; sync model-rotation.yaml with scripts/model_router.py limits (noted in Devin analysis).

devin-ai-integration

Devin Review found 4 new potential issues.

devin-ai-integration · 2026-08-10T03:03:56Z

+        if "GITHUB_OUTPUT" in os.environ:
+            try:
+                with open(os.environ["GITHUB_OUTPUT"], "a") as go:
+                    go.write("provider=none\n")
+                    go.write("model=\n")
+                    go.write("skip=true\n")
+                    go.write(f"reason=Model Router crashed: {e}\n")
+            except Exception:
+                pass


🟡 Model selection step can hard-fail when the router hits an unexpected error

The router's emergency "no capacity" result is appended as raw text (go.write(f"reason=Model Router crashed: {e}\n") at scripts/model_router.py:445) without the multi-line-safe delimiter format, so an error message that spans more than one line makes the whole step fail instead of degrading gracefully.
Impact: Instead of quietly skipping AI triage/review, the workflow job errors out, so pull requests and issues get a red check.

Mechanism: GITHUB_OUTPUT key=value lines must be single-line or use a heredoc delimiter

The whole point of the top-level except in scripts/model_router.py:429-447 is graceful degradation. But GitHub Actions parses $GITHUB_OUTPUT line-by-line as key=value; any embedded newline in the value produces Error: Unable to process file command 'output' successfully. Invalid format, which fails the step (the composite step runs with bash -e and the runner reports the file-command error). Exception messages with newlines are entirely possible (e.g. json.JSONDecodeError-style or OSError text carrying embedded newlines from external data).

The safe pattern is the heredoc form already used elsewhere in the repo (.github/workflows/gemini-triage.yml:95-108 uses echo "body<<EOF" ... EOF), or simply sanitising newlines out of the reason string before writing.

Suggested change

if "GITHUB_OUTPUT" in os.environ:

try:

with open(os.environ["GITHUB_OUTPUT"], "a") as go:

go.write("provider=none\n")

go.write("model=\n")

go.write("skip=true\n")

go.write(f"reason=Model Router crashed: {e}\n")

except Exception:

pass

if "GITHUB_OUTPUT" in os.environ:

try:

safe_reason = f"Model Router crashed: {e}".replace("\r", " ").replace("\n", " ")

with open(os.environ["GITHUB_OUTPUT"], "a") as go:

go.write("provider=none\n")

go.write("model=\n")

go.write("skip=true\n")

go.write(f"reason={safe_reason}\n")

except Exception:

pass

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-10T03:03:57Z

+        # Cross-reference OpenRouter polled availability
+        if provider == "openrouter":
+            if polled_free_models is not None:
+                if model not in polled_free_models:
+                    sys.stderr.write(f"Warning: OpenRouter model {model} is not currently available/free in polled catalog. Skipping.\n")
+                    continue
+            else:
+                # No catalog (fresh or cached) available at all! Skip unverified new models conservatively.
+                if model not in LEGACY_MODELS:
+                    sys.stderr.write(f"Warning: No OpenRouter catalog available. Skipping unverified new model '{model}' conservatively.\n")
+                    continue


📝 Info: Conservative skip when the OpenRouter catalog is unreachable can starve the peer path

When neither a fresh nor a stale catalog is available, only ids in LEGACY_MODELS are allowed (scripts/model_router.py:326-330). For the review role the top-scored candidates cohere/north-mini-code:free and google/gemma-4-31b-it:free are not in that set, so a transient openrouter.ai outage silently pushes routing onto the legacy models and then onto the 15/day Gemini residuals. The behaviour is intentional and safe, but the stderr warnings are not surfaced as GitHub annotations (::warning::), so operators won't see why routing changed.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-10T03:03:58Z

+    original_parse_yaml = mr.parse_yaml
+    monkeypatch.setattr(mr, "parse_yaml", lambda path: original_parse_yaml(str(matrix_file)) if "success" in path else {})


📝 Info: Test monkeypatches parse_yaml with a substring heuristic

monkeypatch.setattr(mr, "parse_yaml", lambda path: original_parse_yaml(str(matrix_file)) if "success" in path else {}) couples the test to the literal substring 'success' appearing in the production path constant (scripts/model_router.py:248). Renaming the schema file would make the test silently exercise an empty matrix (all scores default to 1000) instead of failing loudly. Making the schema path an injectable parameter/env var would remove the need for this patching entirely.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-08-10T03:03:59Z

+            } catch (err) {
+              const msg = err.message || '';
+              if (msg.includes('usage limit exceeded') || msg.includes('USAGE_LIMIT_EXCEEDED')) {
+                core.warning('Linear workspace limit exceeded — skipping sync cleanly: ' + msg);
+              } else {
+                throw err;
              }
            }


📝 Info: Linear sync now swallows only string-matched usage-limit errors

The new top-level catch keys off msg.includes('usage limit exceeded') || msg.includes('USAGE_LIMIT_EXCEEDED') against err.message, but linear() throws new Error(JSON.stringify(json.errors)) (see the helper defined above at .github/workflows/agent-feedback-linear-sync.yml:85-87), so the match depends on Linear's exact GraphQL error payload wording remaining stable. If Linear changes the message (or returns the limit as an extensions.code only), the workflow will go back to failing the job. Matching on a broader signal (e.g. case-insensitive 'usage limit' / 'limit exceeded') would be more robust.

Was this helpful? React with 👍 or 👎 to provide feedback.

github-actions · 2026-08-10T03:04:06Z

head_sha: 15391b0
ready: true
autofix_requested: false
timed_out: false

Peer review gate (ready for second-pass agents)

External reviewers polled: CodeRabbit, Devin, Aikido, Sentry, Copilot.
Autofix (if any) was requested in a separate comment on this SHA.

Peer activity (truncated):

review @devin-ai-integration[bot] state=COMMENTED sha=15391b0

Downstream: gemini-after-peers. Jules: agent-review-auto-jules.

…Seek hold Implements: RL-05 Related: #123 #134

…Seek hold (#136) docs(rate-limit-rotation): RL-05 executing after #123; add RL-18 DeepSeek hold

github-project-automation Bot added this to @timerloggedout-spec's untitled project team planning Aug 9, 2026

github-project-automation Bot moved this to Todo in @timerloggedout-spec's untitled project team planning Aug 9, 2026

google-labs-jules Bot mentioned this pull request Aug 9, 2026

Model Availability Polling #122

Open

vercel Bot deployed to Preview August 9, 2026 21:53 View deployment

This comment was marked as resolved.

Sign in to view

This comment was marked as off-topic.

Sign in to view

vercel Bot deployed to Preview August 9, 2026 21:58 View deployment

This comment was marked as resolved.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Update scripts/model_router.py

f10d545

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>

vercel Bot deployed to Preview August 9, 2026 22:20 View deployment

vercel Bot deployed to Preview August 9, 2026 22:25 View deployment

vercel Bot deployed to Preview August 9, 2026 22:29 View deployment

timerloggedout-spec mentioned this pull request Aug 9, 2026

WorkFlows @ call; received & wait/pending notifications. #124

Open

This comment was marked as resolved.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as resolved.

Sign in to view

vercel Bot deployed to Preview August 10, 2026 02:57 View deployment

timerloggedout-spec approved these changes Aug 10, 2026

View reviewed changes

timerloggedout-spec merged commit b031cbf into master Aug 10, 2026
14 of 16 checks passed

devin-ai-integration Bot reviewed Aug 10, 2026

View reviewed changes

timerloggedout-spec added a commit that referenced this pull request Aug 10, 2026

docs(rate-limit-rotation): RL-05 executing after #123; add RL-18 Deep…

439bc48

…Seek hold Implements: RL-05 Related: #123 #134

timerloggedout-spec mentioned this pull request Aug 10, 2026

docs(rate-limit-rotation): RL-05 executing after #123; add RL-18 DeepSeek hold #136

Merged

timerloggedout-spec added a commit that referenced this pull request Aug 10, 2026

docs(rate-limit-rotation): RL-05 executing after #123; add RL-18 Deep…

60a4f6b

…Seek hold (#136) docs(rate-limit-rotation): RL-05 executing after #123; add RL-18 DeepSeek hold

timerloggedout-spec mentioned this pull request Aug 10, 2026

feat(ci): integrate DeepSeek v4-Pro CI with peer routing, caching, and security controls #134

Closed

devin-ai-integration Bot mentioned this pull request Aug 10, 2026

Integrate MCP Agent Mail Coordination Layer in GitHub Actions #143

Open

timerloggedout-spec mentioned this pull request Aug 24, 2026

gh-Actions Continuous Evaluation #337

Open

qodo-code-review Bot mentioned this pull request Aug 24, 2026

feat(telemetry): Projects + Discussions collectors; agentic-report L0 canary dry-run #351

Merged

		original_parse_yaml = mr.parse_yaml
		monkeypatch.setattr(mr, "parse_yaml", lambda path: original_parse_yaml(str(matrix_file)) if "success" in path else {})

Conversation

google-labs-jules Bot commented Aug 9, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-labs-jules Bot commented Aug 9, 2026

Uh oh!

vercel Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blocksorg Bot commented Aug 9, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Aug 9, 2026

Peer review gate (ready for second-pass agents)

Uh oh!

github-actions Bot commented Aug 9, 2026

Feedback excerpt

Instructions

Uh oh!

This comment was marked as off-topic.

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Aug 9, 2026

Peer review gate (ready for second-pass agents)

Uh oh!

This comment was marked as off-topic.

timerloggedout-spec commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026

Peer review gate (ready for second-pass agents)

Uh oh!

github-actions Bot commented Aug 9, 2026

Feedback excerpt

Instructions

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Aug 9, 2026

🔀 OpenRouter review (qwen/qwen3-coder:free)

Uh oh!

This comment was marked as off-topic.

This comment was marked as resolved.

Uh oh!

timerloggedout-spec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

timerloggedout-spec commented Aug 10, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 10, 2026

Peer review gate (ready for second-pass agents)

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

google-labs-jules Bot commented Aug 9, 2026 •

edited by devin-ai-integration Bot

Loading

vercel Bot commented Aug 9, 2026 •

edited

Loading

🔀 OpenRouter review (`qwen/qwen3-coder:free`)